You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Jed Glazner <jg...@beyondoblivion.com> on 2011/04/27 21:51:43 UTC
Replicaiton Fails with Unreachable error when master host is responding.
Hello All,
I'm having a very strange problem that I just can't figure out. The
slave is not able to replicate from the master, even though the master
is reachable from the slave machine. I can telnet to the port it's
running on, I can use text based browsers to navigate the master from
the slave. I just don't understand why it won't replicate. The admin
screen gives me an Unreachable in the status, and in the log there is an
exception thrown. Details below:
BACKGROUND:
OS: Arch Linux
Solr Version: svn revision 1096983 from
https://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/
No custom plugins, just whatever came with the version above.
Java Setup:
java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10) (ArchLinux-6.b22_1.10-1-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
We have 3 cores running, all 3 cores are not able to replicate.
The admin on the slave shows the Master as
http://solr-master-01_dev.la.bo:8983/solr/music/replication - *Unreachable*
Replicaiton def on the slave
529 <requestHandler name="/replication" class="solr.ReplicationHandler" >
530 <lst name="${slave:slave}">
531 <str
name="masterUrl">http://solr-master-01_dev.la.bo:8983/solr/music/replication</str>
532 <str name="pollInterval">00:15:00</str>
533 </lst>
534 </requestHandler>
Replication def on the master:
529 <requestHandler name="/replication" class="solr.ReplicationHandler" >
530 <lst name="${master:master}">
531 <str name="replicateAfter">commit</str>
532 <str name="replicateAfter">startup</str>
533 <str name="confFiles">schema.xml,stopwords.txt</str>
534 </lst>
535 </requestHandler>
Below is the log start to finish for replication attempts, note that it
says connection refused, however, I can telnet to 8983 from the slave to
the master, so I know it's up and reachable from the slave:
telnet solr-master-01_dev.la.bo 8983
Trying 172.12.65.58...
Connected to solr-master-01_dev.la.bo.
Escape character is '^]'.
I double checked the master to make sure that it didn't have replication
turned off, and it's not. So I should be able to replicate but it
can't. I just dont' know what else to check. The log from the slave is
below.
Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse <init>
WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
use the corresponding class in org.apache.solr.response
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.solr.handler.ReplicationHandler
getReplicationDetails
WARNING: Exception while invoking 'details' method for replication on
master
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
at
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at
org.apache.solr.handler.SnapPuller.getNamedListResponse(SnapPuller.java:193)
at
org.apache.solr.handler.SnapPuller.getCommandResponse(SnapPuller.java:188)
at
org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:588)
at
org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:183)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360)
at
org.apache.jsp.admin.replication.index_jsp.executeCommand(org.apache.jsp.admin.replication.index_jsp:50)
at
org.apache.jsp.admin.replication.index_jsp._jspService(org.apache.jsp.admin.replication.index_jsp:232)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
INFO: [music] webapp=null path=null params={command=details} status=0
QTime=66
Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse <init>
WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
use the corresponding class in org.apache.solr.response
Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
INFO: [music] webapp=null path=null params={command=fetchindex} status=0
QTime=1
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
SEVERE: Master at:
http://solr-master-01_dev.la.bo:8983/solr/music/replication is not
available. Index fetch failed. Exception: Connection refused
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:40:00 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
SEVERE: Master at: http://solr-master-01_dev:8983/solr/rovi/replication
is not available. Index fetch failed. Exception: Connection refused
Re: Replicaiton Fails with Unreachable error when master host is responding.
Posted by Erick Erickson <er...@gmail.com>.
thanks for bringing closure here. Problems like this drive me crazy,
especially when the solution is really simple, but hard to figure out!
Erick
On Wed, May 4, 2011 at 1:14 AM, Jed Glazner <jg...@beyondoblivion.com> wrote:
> So it turns out that it's the host names. According the DNS RFC underscores
> are not valid in host names. Most DNS servers now support them, but it's not
> in the rfc strictly speaking. So there must be something in the underlying
> java classes that bork when using underscores in host names, though I
> didn't see anything in the stack trace that indicated an invalid host name
> exception. That was most the issue though. Once I changed the host name to
> the master's IP address replication worked great. So I'm working with our
> IT to remove underscores from our host names.
>
> Just thought I would post my answer here in case anyone else had that issue.
>
> Thanks.
>
> Jed.
>
> On 04/28/2011 02:03 PM, Mike Sokolov wrote:
>>
>> No clue. Try wireshark to gather more data?
>>
>> On 04/28/2011 02:53 PM, Jed Glazner wrote:
>>>
>>> Anybody?
>>>
>>> On 04/27/2011 01:51 PM, Jed Glazner wrote:
>>>>
>>>> Hello All,
>>>>
>>>> I'm having a very strange problem that I just can't figure out. The
>>>> slave is not able to replicate from the master, even though the master
>>>> is reachable from the slave machine. I can telnet to the port it's
>>>> running on, I can use text based browsers to navigate the master from
>>>> the slave. I just don't understand why it won't replicate. The admin
>>>> screen gives me an Unreachable in the status, and in the log there is an
>>>> exception thrown. Details below:
>>>>
>>>> BACKGROUND:
>>>>
>>>> OS: Arch Linux
>>>> Solr Version: svn revision 1096983 from
>>>> https://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/
>>>> No custom plugins, just whatever came with the version above.
>>>> Java Setup:
>>>>
>>>> java version "1.6.0_22"
>>>> OpenJDK Runtime Environment (IcedTea6 1.10)
>>>> (ArchLinux-6.b22_1.10-1-x86_64)
>>>> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
>>>>
>>>> We have 3 cores running, all 3 cores are not able to replicate.
>>>>
>>>> The admin on the slave shows the Master as
>>>> http://solr-master-01_dev.la.bo:8983/solr/music/replication -
>>>> *Unreachable*
>>>> Replicaiton def on the slave
>>>>
>>>> 529<requestHandler name="/replication"
>>>> class="solr.ReplicationHandler">
>>>> 530<lst name="${slave:slave}">
>>>> 531<str
>>>>
>>>> name="masterUrl">http://solr-master-01_dev.la.bo:8983/solr/music/replication</str>
>>>> 532<str name="pollInterval">00:15:00</str>
>>>> 533</lst>
>>>> 534</requestHandler>
>>>>
>>>> Replication def on the master:
>>>>
>>>> 529<requestHandler name="/replication"
>>>> class="solr.ReplicationHandler">
>>>> 530<lst name="${master:master}">
>>>> 531<str name="replicateAfter">commit</str>
>>>> 532<str name="replicateAfter">startup</str>
>>>> 533<str name="confFiles">schema.xml,stopwords.txt</str>
>>>> 534</lst>
>>>> 535</requestHandler>
>>>>
>>>> Below is the log start to finish for replication attempts, note that it
>>>> says connection refused, however, I can telnet to 8983 from the slave to
>>>> the master, so I know it's up and reachable from the slave:
>>>>
>>>> telnet solr-master-01_dev.la.bo 8983
>>>> Trying 172.12.65.58...
>>>> Connected to solr-master-01_dev.la.bo.
>>>> Escape character is '^]'.
>>>>
>>>> I double checked the master to make sure that it didn't have replication
>>>> turned off, and it's not. So I should be able to replicate but it
>>>> can't. I just dont' know what else to check. The log from the slave is
>>>> below.
>>>>
>>>> Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse<init>
>>>> WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
>>>> use the corresponding class in org.apache.solr.response
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>>> request: Connection refused
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: Retrying request
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>>> request: Connection refused
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: Retrying request
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>>> request: Connection refused
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: Retrying request
>>>> Apr 27, 2011 7:39:45 PM org.apache.solr.handler.ReplicationHandler
>>>> getReplicationDetails
>>>> WARNING: Exception while invoking 'details' method for replication on
>>>> master
>>>> java.net.ConnectException: Connection refused
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at
>>>>
>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>>>> at
>>>>
>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>>>> at
>>>>
>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>>> at java.net.Socket.connect(Socket.java:546)
>>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>>> at
>>>>
>>>> org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
>>>> at
>>>>
>>>> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
>>>> at
>>>>
>>>> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
>>>> at
>>>>
>>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
>>>> at
>>>>
>>>> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
>>>> at
>>>>
>>>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
>>>> at
>>>>
>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
>>>> at
>>>>
>>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
>>>> at
>>>>
>>>> org.apache.solr.handler.SnapPuller.getNamedListResponse(SnapPuller.java:193)
>>>> at
>>>>
>>>> org.apache.solr.handler.SnapPuller.getCommandResponse(SnapPuller.java:188)
>>>> at
>>>>
>>>> org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:588)
>>>> at
>>>>
>>>> org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:183)
>>>> at
>>>>
>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360)
>>>> at
>>>>
>>>> org.apache.jsp.admin.replication.index_jsp.executeCommand(org.apache.jsp.admin.replication.index_jsp:50)
>>>> at
>>>>
>>>> org.apache.jsp.admin.replication.index_jsp._jspService(org.apache.jsp.admin.replication.index_jsp:232)
>>>> at
>>>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>> at
>>>>
>>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
>>>> at
>>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
>>>> at
>>>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>>> at
>>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>>> at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>>>> at
>>>>
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>> at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>> at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>> at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>> at
>>>> org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
>>>> at
>>>> org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
>>>> at
>>>>
>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275)
>>>> at
>>>>
>>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>>> at
>>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>>> at
>>>>
>>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>>> at
>>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>>> at
>>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>>> at
>>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>>> at
>>>>
>>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>>> at
>>>>
>>>> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>>>> at
>>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>>> at org.mortbay.jetty.Server.handle(Server.java:326)
>>>> at
>>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>>> at
>>>>
>>>> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
>>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
>>>> at
>>>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>>>> at
>>>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>>> at
>>>>
>>>> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>>>> at
>>>>
>>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>>> Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
>>>> INFO: [music] webapp=null path=null params={command=details} status=0
>>>> QTime=66
>>>> Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse<init>
>>>> WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
>>>> use the corresponding class in org.apache.solr.response
>>>> Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
>>>> INFO: [music] webapp=null path=null params={command=fetchindex} status=0
>>>> QTime=1
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>>> request: Connection refused
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: Retrying request
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>>> request: Connection refused
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: Retrying request
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>>> request: Connection refused
>>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: Retrying request
>>>> Apr 27, 2011 7:39:45 PM org.apache.solr.handler.SnapPuller
>>>> fetchLatestIndex
>>>> SEVERE: Master at:
>>>> http://solr-master-01_dev.la.bo:8983/solr/music/replication is not
>>>> available. Index fetch failed. Exception: Connection refused
>>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>>> request: Connection refused
>>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: Retrying request
>>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>>> request: Connection refused
>>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: Retrying request
>>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>>> request: Connection refused
>>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>>> executeWithRetry
>>>> INFO: Retrying request
>>>> Apr 27, 2011 7:40:00 PM org.apache.solr.handler.SnapPuller
>>>> fetchLatestIndex
>>>> SEVERE: Master at:http://solr-master-01_dev:8983/solr/rovi/replication
>>>> is not available. Index fetch failed. Exception: Connection refused
>>>>
>>>>
>>>
>>> --
>>> Jed Glazner (801-360-0181)
>>>
>>> This email and its attachments (if any) are for the sole use of the
>>> intended recipient, and may contain private, confidential, and
>>> privileged material. Any review, copying, or distribution of this
>>> email, its attachments or the information contained herein is strictly
>>> prohibited. If you are not the intended recipient, please contact the
>>> sender immediately and permanently delete the original and any copies
>>> of this email and any attachments.
>>>
>
>
Re: Replicaiton Fails with Unreachable error when master host is
responding.
Posted by Jed Glazner <jg...@beyondoblivion.com>.
So it turns out that it's the host names. According the DNS RFC
underscores are not valid in host names. Most DNS servers now support
them, but it's not in the rfc strictly speaking. So there must be
something in the underlying java classes that bork when using
underscores in host names, though I didn't see anything in the stack
trace that indicated an invalid host name exception. That was most the
issue though. Once I changed the host name to the master's IP address
replication worked great. So I'm working with our IT to remove
underscores from our host names.
Just thought I would post my answer here in case anyone else had that
issue.
Thanks.
Jed.
On 04/28/2011 02:03 PM, Mike Sokolov wrote:
> No clue. Try wireshark to gather more data?
>
> On 04/28/2011 02:53 PM, Jed Glazner wrote:
>> Anybody?
>>
>> On 04/27/2011 01:51 PM, Jed Glazner wrote:
>>> Hello All,
>>>
>>> I'm having a very strange problem that I just can't figure out. The
>>> slave is not able to replicate from the master, even though the master
>>> is reachable from the slave machine. I can telnet to the port it's
>>> running on, I can use text based browsers to navigate the master from
>>> the slave. I just don't understand why it won't replicate. The admin
>>> screen gives me an Unreachable in the status, and in the log there is an
>>> exception thrown. Details below:
>>>
>>> BACKGROUND:
>>>
>>> OS: Arch Linux
>>> Solr Version: svn revision 1096983 from
>>> https://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/
>>> No custom plugins, just whatever came with the version above.
>>> Java Setup:
>>>
>>> java version "1.6.0_22"
>>> OpenJDK Runtime Environment (IcedTea6 1.10) (ArchLinux-6.b22_1.10-1-x86_64)
>>> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
>>>
>>> We have 3 cores running, all 3 cores are not able to replicate.
>>>
>>> The admin on the slave shows the Master as
>>> http://solr-master-01_dev.la.bo:8983/solr/music/replication - *Unreachable*
>>> Replicaiton def on the slave
>>>
>>> 529<requestHandler name="/replication" class="solr.ReplicationHandler">
>>> 530<lst name="${slave:slave}">
>>> 531<str
>>> name="masterUrl">http://solr-master-01_dev.la.bo:8983/solr/music/replication</str>
>>> 532<str name="pollInterval">00:15:00</str>
>>> 533</lst>
>>> 534</requestHandler>
>>>
>>> Replication def on the master:
>>>
>>> 529<requestHandler name="/replication" class="solr.ReplicationHandler">
>>> 530<lst name="${master:master}">
>>> 531<str name="replicateAfter">commit</str>
>>> 532<str name="replicateAfter">startup</str>
>>> 533<str name="confFiles">schema.xml,stopwords.txt</str>
>>> 534</lst>
>>> 535</requestHandler>
>>>
>>> Below is the log start to finish for replication attempts, note that it
>>> says connection refused, however, I can telnet to 8983 from the slave to
>>> the master, so I know it's up and reachable from the slave:
>>>
>>> telnet solr-master-01_dev.la.bo 8983
>>> Trying 172.12.65.58...
>>> Connected to solr-master-01_dev.la.bo.
>>> Escape character is '^]'.
>>>
>>> I double checked the master to make sure that it didn't have replication
>>> turned off, and it's not. So I should be able to replicate but it
>>> can't. I just dont' know what else to check. The log from the slave is
>>> below.
>>>
>>> Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse<init>
>>> WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
>>> use the corresponding class in org.apache.solr.response
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>> request: Connection refused
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: Retrying request
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>> request: Connection refused
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: Retrying request
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>> request: Connection refused
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: Retrying request
>>> Apr 27, 2011 7:39:45 PM org.apache.solr.handler.ReplicationHandler
>>> getReplicationDetails
>>> WARNING: Exception while invoking 'details' method for replication on
>>> master
>>> java.net.ConnectException: Connection refused
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at
>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>>> at
>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>>> at
>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>>> at java.net.Socket.connect(Socket.java:546)
>>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>> at
>>> org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
>>> at
>>> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
>>> at
>>> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
>>> at
>>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
>>> at
>>> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
>>> at
>>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
>>> at
>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
>>> at
>>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
>>> at
>>> org.apache.solr.handler.SnapPuller.getNamedListResponse(SnapPuller.java:193)
>>> at
>>> org.apache.solr.handler.SnapPuller.getCommandResponse(SnapPuller.java:188)
>>> at
>>> org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:588)
>>> at
>>> org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:183)
>>> at
>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360)
>>> at
>>> org.apache.jsp.admin.replication.index_jsp.executeCommand(org.apache.jsp.admin.replication.index_jsp:50)
>>> at
>>> org.apache.jsp.admin.replication.index_jsp._jspService(org.apache.jsp.admin.replication.index_jsp:232)
>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>> at
>>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
>>> at
>>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
>>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>> at
>>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>> at
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>>> at
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>> at
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>> at
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>> at
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>> at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
>>> at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
>>> at
>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275)
>>> at
>>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>> at
>>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>> at
>>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>> at
>>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>> at
>>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>> at
>>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>> at
>>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>> at
>>> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>>> at
>>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>> at org.mortbay.jetty.Server.handle(Server.java:326)
>>> at
>>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>> at
>>> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>> at
>>> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>>> at
>>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>> Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
>>> INFO: [music] webapp=null path=null params={command=details} status=0
>>> QTime=66
>>> Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse<init>
>>> WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
>>> use the corresponding class in org.apache.solr.response
>>> Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
>>> INFO: [music] webapp=null path=null params={command=fetchindex} status=0
>>> QTime=1
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>> request: Connection refused
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: Retrying request
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>> request: Connection refused
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: Retrying request
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>> request: Connection refused
>>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: Retrying request
>>> Apr 27, 2011 7:39:45 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
>>> SEVERE: Master at:
>>> http://solr-master-01_dev.la.bo:8983/solr/music/replication is not
>>> available. Index fetch failed. Exception: Connection refused
>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>> request: Connection refused
>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: Retrying request
>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>> request: Connection refused
>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: Retrying request
>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: I/O exception (java.net.ConnectException) caught when processing
>>> request: Connection refused
>>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>>> executeWithRetry
>>> INFO: Retrying request
>>> Apr 27, 2011 7:40:00 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
>>> SEVERE: Master at:http://solr-master-01_dev:8983/solr/rovi/replication
>>> is not available. Index fetch failed. Exception: Connection refused
>>>
>>>
>>
>> --
>> Jed Glazner (801-360-0181)
>>
>> This email and its attachments (if any) are for the sole use of the
>> intended recipient, and may contain private, confidential, and
>> privileged material. Any review, copying, or distribution of this
>> email, its attachments or the information contained herein is strictly
>> prohibited. If you are not the intended recipient, please contact the
>> sender immediately and permanently delete the original and any copies
>> of this email and any attachments.
>>
Re: Replicaiton Fails with Unreachable error when master host is
responding.
Posted by Mike Sokolov <so...@ifactory.com>.
No clue. Try wireshark to gather more data?
On 04/28/2011 02:53 PM, Jed Glazner wrote:
> Anybody?
>
> On 04/27/2011 01:51 PM, Jed Glazner wrote:
>> Hello All,
>>
>> I'm having a very strange problem that I just can't figure out. The
>> slave is not able to replicate from the master, even though the master
>> is reachable from the slave machine. I can telnet to the port it's
>> running on, I can use text based browsers to navigate the master from
>> the slave. I just don't understand why it won't replicate. The admin
>> screen gives me an Unreachable in the status, and in the log there is an
>> exception thrown. Details below:
>>
>> BACKGROUND:
>>
>> OS: Arch Linux
>> Solr Version: svn revision 1096983 from
>> https://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/
>> No custom plugins, just whatever came with the version above.
>> Java Setup:
>>
>> java version "1.6.0_22"
>> OpenJDK Runtime Environment (IcedTea6 1.10) (ArchLinux-6.b22_1.10-1-x86_64)
>> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
>>
>> We have 3 cores running, all 3 cores are not able to replicate.
>>
>> The admin on the slave shows the Master as
>> http://solr-master-01_dev.la.bo:8983/solr/music/replication - *Unreachable*
>> Replicaiton def on the slave
>>
>> 529<requestHandler name="/replication" class="solr.ReplicationHandler">
>> 530<lst name="${slave:slave}">
>> 531<str
>> name="masterUrl">http://solr-master-01_dev.la.bo:8983/solr/music/replication</str>
>> 532<str name="pollInterval">00:15:00</str>
>> 533</lst>
>> 534</requestHandler>
>>
>> Replication def on the master:
>>
>> 529<requestHandler name="/replication" class="solr.ReplicationHandler">
>> 530<lst name="${master:master}">
>> 531<str name="replicateAfter">commit</str>
>> 532<str name="replicateAfter">startup</str>
>> 533<str name="confFiles">schema.xml,stopwords.txt</str>
>> 534</lst>
>> 535</requestHandler>
>>
>> Below is the log start to finish for replication attempts, note that it
>> says connection refused, however, I can telnet to 8983 from the slave to
>> the master, so I know it's up and reachable from the slave:
>>
>> telnet solr-master-01_dev.la.bo 8983
>> Trying 172.12.65.58...
>> Connected to solr-master-01_dev.la.bo.
>> Escape character is '^]'.
>>
>> I double checked the master to make sure that it didn't have replication
>> turned off, and it's not. So I should be able to replicate but it
>> can't. I just dont' know what else to check. The log from the slave is
>> below.
>>
>> Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse<init>
>> WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
>> use the corresponding class in org.apache.solr.response
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: I/O exception (java.net.ConnectException) caught when processing
>> request: Connection refused
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: Retrying request
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: I/O exception (java.net.ConnectException) caught when processing
>> request: Connection refused
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: Retrying request
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: I/O exception (java.net.ConnectException) caught when processing
>> request: Connection refused
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: Retrying request
>> Apr 27, 2011 7:39:45 PM org.apache.solr.handler.ReplicationHandler
>> getReplicationDetails
>> WARNING: Exception while invoking 'details' method for replication on
>> master
>> java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>> at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
>> at java.net.Socket.connect(Socket.java:546)
>> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at
>> org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
>> at
>> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
>> at
>> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
>> at
>> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
>> at
>> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
>> at
>> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
>> at
>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
>> at
>> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
>> at
>> org.apache.solr.handler.SnapPuller.getNamedListResponse(SnapPuller.java:193)
>> at
>> org.apache.solr.handler.SnapPuller.getCommandResponse(SnapPuller.java:188)
>> at
>> org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:588)
>> at
>> org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:183)
>> at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360)
>> at
>> org.apache.jsp.admin.replication.index_jsp.executeCommand(org.apache.jsp.admin.replication.index_jsp:50)
>> at
>> org.apache.jsp.admin.replication.index_jsp._jspService(org.apache.jsp.admin.replication.index_jsp:232)
>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> at
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
>> at
>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
>> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>> at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
>> at
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>> at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> at
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>> at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
>> at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
>> at
>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275)
>> at
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>> at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>> at
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>> at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> at
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>> at
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> at
>> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>> at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> at org.mortbay.jetty.Server.handle(Server.java:326)
>> at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>> at
>> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>> at
>> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
>> at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
>> INFO: [music] webapp=null path=null params={command=details} status=0
>> QTime=66
>> Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse<init>
>> WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
>> use the corresponding class in org.apache.solr.response
>> Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
>> INFO: [music] webapp=null path=null params={command=fetchindex} status=0
>> QTime=1
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: I/O exception (java.net.ConnectException) caught when processing
>> request: Connection refused
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: Retrying request
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: I/O exception (java.net.ConnectException) caught when processing
>> request: Connection refused
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: Retrying request
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: I/O exception (java.net.ConnectException) caught when processing
>> request: Connection refused
>> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: Retrying request
>> Apr 27, 2011 7:39:45 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
>> SEVERE: Master at:
>> http://solr-master-01_dev.la.bo:8983/solr/music/replication is not
>> available. Index fetch failed. Exception: Connection refused
>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: I/O exception (java.net.ConnectException) caught when processing
>> request: Connection refused
>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: Retrying request
>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: I/O exception (java.net.ConnectException) caught when processing
>> request: Connection refused
>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: Retrying request
>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: I/O exception (java.net.ConnectException) caught when processing
>> request: Connection refused
>> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
>> executeWithRetry
>> INFO: Retrying request
>> Apr 27, 2011 7:40:00 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
>> SEVERE: Master at:http://solr-master-01_dev:8983/solr/rovi/replication
>> is not available. Index fetch failed. Exception: Connection refused
>>
>>
>
>
> --
> Jed Glazner (801-360-0181)
>
> This email and its attachments (if any) are for the sole use of the
> intended recipient, and may contain private, confidential, and
> privileged material. Any review, copying, or distribution of this
> email, its attachments or the information contained herein is strictly
> prohibited. If you are not the intended recipient, please contact the
> sender immediately and permanently delete the original and any copies
> of this email and any attachments.
>
Re: Replicaiton Fails with Unreachable error when master host is
responding.
Posted by Jed Glazner <jg...@beyondoblivion.com>.
Anybody?
On 04/27/2011 01:51 PM, Jed Glazner wrote:
> Hello All,
>
> I'm having a very strange problem that I just can't figure out. The
> slave is not able to replicate from the master, even though the master
> is reachable from the slave machine. I can telnet to the port it's
> running on, I can use text based browsers to navigate the master from
> the slave. I just don't understand why it won't replicate. The admin
> screen gives me an Unreachable in the status, and in the log there is an
> exception thrown. Details below:
>
> BACKGROUND:
>
> OS: Arch Linux
> Solr Version: svn revision 1096983 from
> https://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/
> No custom plugins, just whatever came with the version above.
> Java Setup:
>
> java version "1.6.0_22"
> OpenJDK Runtime Environment (IcedTea6 1.10) (ArchLinux-6.b22_1.10-1-x86_64)
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
>
> We have 3 cores running, all 3 cores are not able to replicate.
>
> The admin on the slave shows the Master as
> http://solr-master-01_dev.la.bo:8983/solr/music/replication - *Unreachable*
> Replicaiton def on the slave
>
> 529<requestHandler name="/replication" class="solr.ReplicationHandler">
> 530<lst name="${slave:slave}">
> 531<str
> name="masterUrl">http://solr-master-01_dev.la.bo:8983/solr/music/replication</str>
> 532<str name="pollInterval">00:15:00</str>
> 533</lst>
> 534</requestHandler>
>
> Replication def on the master:
>
> 529<requestHandler name="/replication" class="solr.ReplicationHandler">
> 530<lst name="${master:master}">
> 531<str name="replicateAfter">commit</str>
> 532<str name="replicateAfter">startup</str>
> 533<str name="confFiles">schema.xml,stopwords.txt</str>
> 534</lst>
> 535</requestHandler>
>
> Below is the log start to finish for replication attempts, note that it
> says connection refused, however, I can telnet to 8983 from the slave to
> the master, so I know it's up and reachable from the slave:
>
> telnet solr-master-01_dev.la.bo 8983
> Trying 172.12.65.58...
> Connected to solr-master-01_dev.la.bo.
> Escape character is '^]'.
>
> I double checked the master to make sure that it didn't have replication
> turned off, and it's not. So I should be able to replicate but it
> can't. I just dont' know what else to check. The log from the slave is
> below.
>
> Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse<init>
> WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
> use the corresponding class in org.apache.solr.response
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: I/O exception (java.net.ConnectException) caught when processing
> request: Connection refused
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: Retrying request
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: I/O exception (java.net.ConnectException) caught when processing
> request: Connection refused
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: Retrying request
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: I/O exception (java.net.ConnectException) caught when processing
> request: Connection refused
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: Retrying request
> Apr 27, 2011 7:39:45 PM org.apache.solr.handler.ReplicationHandler
> getReplicationDetails
> WARNING: Exception while invoking 'details' method for replication on
> master
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
> at java.net.Socket.connect(Socket.java:546)
> at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
> at
> org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
> at
> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
> at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> at
> org.apache.solr.handler.SnapPuller.getNamedListResponse(SnapPuller.java:193)
> at
> org.apache.solr.handler.SnapPuller.getCommandResponse(SnapPuller.java:188)
> at
> org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:588)
> at
> org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:183)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360)
> at
> org.apache.jsp.admin.replication.index_jsp.executeCommand(org.apache.jsp.admin.replication.index_jsp:50)
> at
> org.apache.jsp.admin.replication.index_jsp._jspService(org.apache.jsp.admin.replication.index_jsp:232)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
> at
> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
> at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
> at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
> INFO: [music] webapp=null path=null params={command=details} status=0
> QTime=66
> Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse<init>
> WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
> use the corresponding class in org.apache.solr.response
> Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
> INFO: [music] webapp=null path=null params={command=fetchindex} status=0
> QTime=1
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: I/O exception (java.net.ConnectException) caught when processing
> request: Connection refused
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: Retrying request
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: I/O exception (java.net.ConnectException) caught when processing
> request: Connection refused
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: Retrying request
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: I/O exception (java.net.ConnectException) caught when processing
> request: Connection refused
> Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: Retrying request
> Apr 27, 2011 7:39:45 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
> SEVERE: Master at:
> http://solr-master-01_dev.la.bo:8983/solr/music/replication is not
> available. Index fetch failed. Exception: Connection refused
> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: I/O exception (java.net.ConnectException) caught when processing
> request: Connection refused
> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: Retrying request
> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: I/O exception (java.net.ConnectException) caught when processing
> request: Connection refused
> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: Retrying request
> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: I/O exception (java.net.ConnectException) caught when processing
> request: Connection refused
> Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
> executeWithRetry
> INFO: Retrying request
> Apr 27, 2011 7:40:00 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
> SEVERE: Master at: http://solr-master-01_dev:8983/solr/rovi/replication
> is not available. Index fetch failed. Exception: Connection refused
>
--
Jed Glazner (801-360-0181)
This email and its attachments (if any) are for the sole use of the
intended recipient, and may contain private, confidential, and
privileged material. Any review, copying, or distribution of this email,
its attachments or the information contained herein is strictly
prohibited. If you are not the intended recipient, please contact the
sender immediately and permanently delete the original and any copies of
this email and any attachments.