You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Jose Nunez <jn...@striketechnologies.com> on 2020/06/07 03:22:15 UTC
Mesos master issues with Zookeper on Fedora 29, RPM distribution:
ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk
retcode=-4,
Hello Mesos experts,
I'm trying to setup a master with two slaves on Fedora 29, using the RPM installation. I do have the following problems:
1. I cannot connect to my master on http://mymaster:5050. I get a connection error
2. Checking the logs I do see the following error message repeating periodically:
/var/log/mesos/mesos-master.WARNING
W0606 23:11:35.007477 63540 authenticator.cpp:513] No credentials provided, authentication requests will be refused
W0606 23:11:45.004431 63539 group.cpp:495] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
W0606 23:11:45.004431 63548 group.cpp:495] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
And using journalctl -xe -u mesos-master:
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@764: Client environment:os.name=Linux
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@765: Client environment:os.arch=XXXX.fc29.x86_64
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@766: Client environment:os.version=#1 SMP Thu Jan 24 14:38:07 EST 2019
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@774: Client environment:user.name=(null)
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@782: Client environment:user.home=/root
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@794: Client environment:user.dir=/
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=mymaster:2181 sessionTimeout=10000 watcher=0x7fb64ebbfa20 sessionId=0 sessionPasswd=<null> context=0x7fb634000ec0 flags=0
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb5f9ffb700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:13:58 mymaster mesos-master[63536]: 2020-06-06 23:13:58,355:63521(0x7fb5fb7fe700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:13:58 mymaster mesos-master[63536]: 2020-06-06 23:13:58,355:63521(0x7fb5fa7fc700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:13:58 mymaster mesos-master[63536]: 2020-06-06 23:13:58,355:63521(0x7fb640dc1700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:13:58 mymaster mesos-master[63536]: 2020-06-06 23:13:58,357:63521(0x7fb5f9ffb700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:14:01 mymaster mesos-master[63536]: 2020-06-06 23:14:01,691:63521(0x7fb5fb7fe700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:14:01 mymaster mesos-master[63536]: 2020-06-06 23:14:01,691:63521(0x7fb5fa7fc700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:14:01 mymaster mesos-master[63536]: 2020-06-06 23:14:01,691:63521(0x7fb640dc1700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:14:01 mymaster mesos-master[63536]: 2020-06-06 23:14:01,693:63521(0x7fb5f9ffb700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:14:05 mymaster mesos-master[63536]: W0606 23:14:05.020342 63552 group.cpp:495] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
Jun 06 23:14:05 mymaster mesos-master[63536]: W0606 23:14:05.020349 63539 group.cpp:495] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
Jun 06 23:14:05 mymaster mesos-master[63536]: W0606 23:14:05.020350 63540 group.cpp:495] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
Jun 06 23:14:05 mymaster mesos-master[63536]: I0606 23:14:05.020401 63552 group.cpp:511] ZooKeeper session expired
Jun 06 23:14:05 mymaster mesos-master[63536]: I0606 23:14:05.020416 63539 group.cpp:511] ZooKeeper session expired
Jun 06 23:14:05 mymaster mesos-master[63536]: I0606 23:14:05.020434 63540 group.cpp:511] ZooKeeper session expired
I have agents running on two different machines and they claim they can see the master. I went through the documentation and I'm not sure what I'm missing.
My '/etc/mesos/zk' has the right URI (not localhost):
zk://mymaster:2181/mesos
Any suggestions?
Thanks!,
--Jose
________________________________
CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments are intended solely for the addressee. This transmission is covered by the Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information contained in this transmission is confidential in nature and protected from further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 (1999), and may be subject to attorney-client or other legal privilege. Your use or disclosure of this information for any purpose other than that intended by its transmittal is strictly prohibited, and may subject you to fines and/or penalties under federal and state law. If you are not the intended recipient of this transmission, please DESTROY ALL COPIES RECEIVED and confirm destruction to the sender via return transmittal.
RE: Mesos master issues with Zookeper on Fedora 29, RPM distribution:
ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk
retcode=-4,
Posted by Jose Nunez <jn...@striketechnologies.com>.
Hello,
I figured this issue. I want to share my findings so new users do not make the same mistakes:
1. Make sure your /etc/hosts resolves the name to the master to a routable IP. Mine was pointing to localhost. So I overrode /etc/mesos-master/ip. Do the same on /etc/mesos/zk for the Zookeeper
2. Zookeeper is not bundled with Mesos. But installation and setup is trivial. There is an excellent guide for system based systems here: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-an-apache-zookeeper-cluster-on-ubuntu-18-04
Hope this helps.
Regards,
--Jose
From: Jose Nunez
Sent: Saturday, June 6, 2020 11:22 PM
To: user@mesos.apache.org
Subject: Mesos master issues with Zookeper on Fedora 29, RPM distribution: ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4,
Hello Mesos experts,
I'm trying to setup a master with two slaves on Fedora 29, using the RPM installation. I do have the following problems:
1. I cannot connect to my master on http://mymaster:5050. I get a connection error
2. Checking the logs I do see the following error message repeating periodically:
/var/log/mesos/mesos-master.WARNING
W0606 23:11:35.007477 63540 authenticator.cpp:513] No credentials provided, authentication requests will be refused
W0606 23:11:45.004431 63539 group.cpp:495] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
W0606 23:11:45.004431 63548 group.cpp:495] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
And using journalctl -xe -u mesos-master:
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@764: Client environment:os.name=Linux
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@765: Client environment:os.arch=XXXX.fc29.x86_64
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@766: Client environment:os.version=#1 SMP Thu Jan 24 14:38:07 EST 2019
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@774: Client environment:user.name=(null)
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@782: Client environment:user.home=/root
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@log_env@794: Client environment:user.dir=/
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb642dc5700):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=mymaster:2181 sessionTimeout=10000 watcher=0x7fb64ebbfa20 sessionId=0 sessionPasswd=<null> context=0x7fb634000ec0 flags=0
Jun 06 23:13:55 mymaster mesos-master[63536]: 2020-06-06 23:13:55,021:63521(0x7fb5f9ffb700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:13:58 mymaster mesos-master[63536]: 2020-06-06 23:13:58,355:63521(0x7fb5fb7fe700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:13:58 mymaster mesos-master[63536]: 2020-06-06 23:13:58,355:63521(0x7fb5fa7fc700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:13:58 mymaster mesos-master[63536]: 2020-06-06 23:13:58,355:63521(0x7fb640dc1700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:13:58 mymaster mesos-master[63536]: 2020-06-06 23:13:58,357:63521(0x7fb5f9ffb700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:14:01 mymaster mesos-master[63536]: 2020-06-06 23:14:01,691:63521(0x7fb5fb7fe700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:14:01 mymaster mesos-master[63536]: 2020-06-06 23:14:01,691:63521(0x7fb5fa7fc700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:14:01 mymaster mesos-master[63536]: 2020-06-06 23:14:01,691:63521(0x7fb640dc1700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:14:01 mymaster mesos-master[63536]: 2020-06-06 23:14:01,693:63521(0x7fb5f9ffb700):ZOO_ERROR@handle_socket_error_msg@1758: Socket [127.0.0.1:2181] zk retcode=-4, errno=111(Connection refused): server refused to accept the client
Jun 06 23:14:05 mymaster mesos-master[63536]: W0606 23:14:05.020342 63552 group.cpp:495] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
Jun 06 23:14:05 mymaster mesos-master[63536]: W0606 23:14:05.020349 63539 group.cpp:495] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
Jun 06 23:14:05 mymaster mesos-master[63536]: W0606 23:14:05.020350 63540 group.cpp:495] Timed out waiting to connect to ZooKeeper. Forcing ZooKeeper session (sessionId=0) expiration
Jun 06 23:14:05 mymaster mesos-master[63536]: I0606 23:14:05.020401 63552 group.cpp:511] ZooKeeper session expired
Jun 06 23:14:05 mymaster mesos-master[63536]: I0606 23:14:05.020416 63539 group.cpp:511] ZooKeeper session expired
Jun 06 23:14:05 mymaster mesos-master[63536]: I0606 23:14:05.020434 63540 group.cpp:511] ZooKeeper session expired
I have agents running on two different machines and they claim they can see the master. I went through the documentation and I'm not sure what I'm missing.
My '/etc/mesos/zk' has the right URI (not localhost):
zk://mymaster:2181/mesos
Any suggestions?
Thanks!,
--Jose
________________________________
CONFIDENTIALITY / PRIVILEGE NOTICE: This transmission and any attachments are intended solely for the addressee. This transmission is covered by the Electronic Communications Privacy Act, 18 U.S.C ''2510-2521. The information contained in this transmission is confidential in nature and protected from further use or disclosure under U.S. Pub. L. 106-102, 113 U.S. Stat. 1338 (1999), and may be subject to attorney-client or other legal privilege. Your use or disclosure of this information for any purpose other than that intended by its transmittal is strictly prohibited, and may subject you to fines and/or penalties under federal and state law. If you are not the intended recipient of this transmission, please DESTROY ALL COPIES RECEIVED and confirm destruction to the sender via return transmittal.