You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "Harald Musum (Jira)" <ji...@apache.org> on 2021/01/19 13:03:00 UTC
[jira] [Created] (ZOOKEEPER-4183) Leader election not working when
using hostname in server config and hostname resolves to an internal IP
addresses
Harald Musum created ZOOKEEPER-4183:
---------------------------------------
Summary: Leader election not working when using hostname in server config and hostname resolves to an internal IP addresses
Key: ZOOKEEPER-4183
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4183
Project: ZooKeeper
Issue Type: Bug
Components: server
Reporter: Harald Musum
We have a working setup with ZooKeeper 3.5.6 with 3 servers and server config similar to:
server.1=1.foo.com:2182:2183:participant
server.2=2.foo.com:2182:2183:participant
server.3=3.foo.com:2182:2183:participan
ZooKeeper servers are running in Docker containers and have IP addresses in the 10.x.x.x range, but also IP addresses from the default Docker network (172.17.x.x addresses) that are only usable inside the Docker
container, /etc/hosts has e.g.:
172.17.2.192 1.foo.com
When upgrading to 3.5.7 leader election failed with the following error in zookeeper log:
.org.apache.zookeeper.server.quorum.QuorumCnxManager Received connection request 10.2.2.192:37028
.org.apache.zookeeper.server.quorum.UnifiedServerSocket Accepted TLS connection from /10.2.2.192:37028 - TLSv1.2 - TLS_ECDHE_RSA_WITH_AES_128_GCM_SH
A256
.org.apache.zookeeper.server.quorum.QuorumCnxManager
Cannot open channel to 0 at election address /172.17.2.192:2183
exception=
java.net.NoRouteToHostException: No route to host (Host unreachable)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:609)
at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:285)
We tried using version 3.6.2, but had the same issue there.
After reading the code and looking at changes between 3.5.6 and 3.5.7 I
found https://issues.apache.org/jira/browse/ZOOKEEPER-3057 and the corresponding PR https://github.com/apache/zookeeper/pull/548
It seems like this PR changed the way election addresses are sent in the message when doing leader election, from using
hostnames (when hostnames are specified in server config) to always using IP addresses.
Patching 3.6.2 with the following change makes this work again for us:
{code:java}
diff --git a/zookeeper-server/zookeeper-server-3.6.2/src/main/java/org/apache/zookeeper/common/NetUtils.java b/zookeeper-server/zookeeper-server-3.6.2/src/main/java/org/apache/zookeeper/common/NetUtils.java
index be8cb9a638..f32f1da7c8 100644
--- a/zookeeper-server/zookeeper-server-3.6.2/src/main/java/org/apache/zookeeper/common/NetUtils.java
+++ b/zookeeper-server/zookeeper-server-3.6.2/src/main/java/org/apache/zookeeper/common/NetUtils.java
@@ -27,13 +27,18 @@ import java.net.InetSocketAddress;
*/
public class NetUtils {+ // Note: Changed from original to use hostname from InetSocketAddress if there exists one
public static String formatInetAddr(InetSocketAddress addr) {
+ String hostName = addr.getHostName();
+ if (hostName != null) {
+ return String.format("%s:%s", hostName, addr.getPort());
+ }
+
InetAddress ia = addr.getAddress(); if (ia == null) {
return String.format("%s:%s", addr.getHostString(), addr.getPort());
} if (ia instanceof Inet6Address) {
return String.format("[%s]:%s", ia.getHostAddress(), addr.getPort());
} else {{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)