You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Anastasios Lisgaras <ta...@yahoo.gr.INVALID> on 2020/03/04 14:44:07 UTC
systemd failed to stop zookeeper-server.
Hello zookeeper community,
After the last system update I had a problem with the zookeeper-server.
The problem was about the same as this:
https://issues.apache.org/jira/browse/BIGTOP-3302
So, I went to the file :
/etc/init.d/zookeeper-server
and I replaced "su" with "runuser". If you want to see it :
# cat /etc/init.d/zookeeper-server : https://termbin.com/gupi
# cat /usr/lib/zookeeper/bin/zkServer.sh : https://termbin.com/3xpb
After this change the server runs flawlessly, but when i try to stop it
via "systemctl" the system (systemd) complains.
```
# systemctl status zookeeper-server
● zookeeper-server.service - LSB: ZooKeeper is a centralized service for
maintaining configuration information, naming, providing distributed
synchronization, and providing group services.
Loaded: loaded (/etc/rc.d/init.d/zookeeper-server; bad; vendor
preset: disabled)
Active: active (running) since Wed 2020-03-04 16:16:51 EET; 2s ago
Docs: man:systemd-sysv-generator(8)
Process: 30509 ExecStart=/etc/rc.d/init.d/zookeeper-server start
(code=exited, status=0/SUCCESS)
Main PID: 30557 (java)
CGroup: /system.slice/zookeeper-server.service
└─30557 /usr/lib/jvm/jre-openjdk/bin/java
-Dzookeeper.datadir.autocreate=false
-Dzookeeper.log.dir=/var/log/zookeeper -Dzookee...
Mar 04 16:16:50 myserver systemd[1]: Starting LSB: ZooKeeper is a
centralized service for maintaining configuration information,...ices....
Mar 04 16:16:50 myserver runuser[30541]: pam_unix(runuser:session):
session opened for user zookeeper by (uid=0)
Mar 04 16:16:50 myserver zookeeper-server[30509]: JMX enabled by default
Mar 04 16:16:50 myserver zookeeper-server[30509]: Using config:
/etc/zookeeper/conf/zoo.cfg
Mar 04 16:16:51 myserver zookeeper-server[30509]: Starting zookeeper ...
STARTED
Mar 04 16:16:51 myserver runuser[30541]: pam_unix(runuser:session):
session closed for user zookeeper
Mar 04 16:16:51 myserver systemd[1]: Started LSB: ZooKeeper is a
centralized service for maintaining configuration information, ...rvices..
Hint: Some lines were ellipsized, use -l to show in full.
# systemctl stop zookeeper-server
# systemctl status zookeeper-server -l
● zookeeper-server.service - LSB: ZooKeeper is a centralized service for
maintaining configuration information, naming, providing distributed
synchronization, and providing group services.
Loaded: loaded (/etc/rc.d/init.d/zookeeper-server; bad; vendor
preset: disabled)
Active: failed (Result: signal) since Wed 2020-03-04 16:17:00 EET; 4s ago
Docs: man:systemd-sysv-generator(8)
Process: 30605 ExecStop=/etc/rc.d/init.d/zookeeper-server stop
(code=exited, status=0/SUCCESS)
Process: 30509 ExecStart=/etc/rc.d/init.d/zookeeper-server start
(code=exited, status=0/SUCCESS)
Main PID: 30557 (code=killed, signal=KILL)
Mar 04 16:16:51 myserver systemd[1]: Started LSB: ZooKeeper is a
centralized service for maintaining configuration information, naming,
providing distributed synchronization, and providing group services..
Mar 04 16:17:00 myserver systemd[1]: Stopping LSB: ZooKeeper is a
centralized service for maintaining configuration information, naming,
providing distributed synchronization, and providing group services....
Mar 04 16:17:00 myserver runuser[30642]: pam_unix(runuser:session):
session opened for user zookeeper by (uid=0)
Mar 04 16:17:00 myserver zookeeper-server[30605]: JMX enabled by default
Mar 04 16:17:00 myserver zookeeper-server[30605]: Using config:
/etc/zookeeper/conf/zoo.cfg
Mar 04 16:17:00 myserver zookeeper-server[30605]: Stopping zookeeper ...
STOPPED
Mar 04 16:17:00 myserver systemd[1]: zookeeper-server.service: main
process exited, code=killed, status=9/KILL
Mar 04 16:17:00 myserver systemd[1]: Stopped LSB: ZooKeeper is a
centralized service for maintaining configuration information, naming,
providing distributed synchronization, and providing group services..
Mar 04 16:17:00 myserver systemd[1]: Unit zookeeper-server.service
entered failed state.
Mar 04 16:17:00 myserver systemd[1]: zookeeper-server.service failed.
# systemctl is-active zookeeper-server
failed
# systemctl is-failed zookeeper-server
failed
# ps -aux | grep zookeeper
root 31181 0.0 0.0 112712 968 pts/0 S+ 16:21 0:00 grep
--color=auto zookeeper
# systemctl restart zookeeper-server -l
# systemctl status zookeeper-server -l
● zookeeper-server.service - LSB: ZooKeeper is a centralized service for
maintaining configuration information, naming, providing distributed
synchronization, and providing group services.
Loaded: loaded (/etc/rc.d/init.d/zookeeper-server; bad; vendor
preset: disabled)
Active: active (running) since Wed 2020-03-04 16:18:37 EET; 2s ago
Docs: man:systemd-sysv-generator(8)
Process: 30605 ExecStop=/etc/rc.d/init.d/zookeeper-server stop
(code=exited, status=0/SUCCESS)
Process: 30746 ExecStart=/etc/rc.d/init.d/zookeeper-server start
(code=exited, status=0/SUCCESS)
Main PID: 30794 (java)
CGroup: /system.slice/zookeeper-server.service
└─30794 /usr/lib/jvm/jre-openjdk/bin/java
-Dzookeeper.datadir.autocreate=false
-Dzookeeper.log.dir=/var/log/zookeeper
-Dzookeeper.root.logger=INFO,ROLLINGFILE -cp
/usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/zookeeper/bin/../lib/jline-2.11.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.16.2.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/*:/usr/lib/zookeeper/lib/*
-Dzookeeper.log.threshold=INFO -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
org.apache.zookeeper.server.quorum.QuorumPeerMain
/etc/zookeeper/conf/zoo.cfg
Mar 04 16:18:35 myserver systemd[1]: Starting LSB: ZooKeeper is a
centralized service for maintaining configuration information, naming,
providing distributed synchronization, and providing group services....
Mar 04 16:18:36 myserver runuser[30778]: pam_unix(runuser:session):
session opened for user zookeeper by (uid=0)
Mar 04 16:18:36 myserver zookeeper-server[30746]: JMX enabled by default
Mar 04 16:18:36 myserver zookeeper-server[30746]: Using config:
/etc/zookeeper/conf/zoo.cfg
Mar 04 16:18:37 myserver zookeeper-server[30746]: Starting zookeeper ...
STARTED
Mar 04 16:18:37 myserver systemd[1]: Started LSB: ZooKeeper is a
centralized service for maintaining configuration information, naming,
providing distributed synchronization, and providing group services..
# ps -aux | grep zookeeper
zookeep+ 30794 1.0 1.8 3499256 72648 ? Sl 16:18 0:02
/usr/lib/jvm/jre-openjdk/bin/java -Dzookeeper.datadir.autocreate=false
-Dzookeeper.log.dir=/var/log/zookeeper
-Dzookeeper.root.logger=INFO,ROLLINGFILE -cp
/usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/zookeeper/bin/../lib/jline-2.11.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.16.2.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/*:/usr/lib/zookeeper/lib/*
-Dzookeeper.log.threshold=INFO -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.local.only=false
org.apache.zookeeper.server.quorum.QuorumPeerMain
/etc/zookeeper/conf/zoo.cfg
root 31105 0.0 0.0 112712 968 pts/0 R+ 16:21 0:00 grep
--color=auto zookeeper
```
As you can see, starting, and restarting works properly.
When I stop the zookeeper-server, then the systemctl in the status
indicates a problem, but the zookeeper-server actually stops.
Can you please help me to fix this problem ?
Because this problem (about systemd ) is relatively recent, the
development team is aware of it ?
OS : CentOS Linux release 7.7.1908 (Core)
# yum info zookeeper-server
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: ftp.ntua.gr
* epel: mirrors.daticum.com
* extras: ftp.ntua.gr
* updates: ftp.ntua.gr
Installed Packages
Name : zookeeper-server
Arch : x86_64
Version : 3.4.5+cdh5.16.2+159
Release : 1.cdh5.16.2.p0.23.el7
Size : 5.0 k
Repo : installed
From repo : cloudera-cdh5
Summary : The Hadoop Zookeeper server
URL : http://zookeeper.apache.org/
License : ASL 2.0
Description : This package starts the zookeeper server on startup
# yum info kafka
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: ftp.ntua.gr
* epel: mirror.hosting90.cz
* extras: ftp.ntua.gr
* updates: ftp.ntua.gr
Installed Packages
Name : kafka
Arch : noarch
Version : 2.2.1+kafka4.1.0
Release : 2.4.1.0.p0.6.el7
Size : 91 M
Repo : installed
From repo : cloudera-kafka
Summary : Apache Kafka is publish-subscribe messaging rethought as a
distributed commit log.
URL : http://kafka.apache.org/
License : ASL 2.0
Description : Apache Kafka is publish-subscribe messaging rethought as a
distributed commit log.
: A single Kafka broker can handle hundreds of megabytes of
reads and writes per second
: from thousands of clients. It can be elastically and
transparently expanded without downtime.
: Data streams are partitioned and spread over a cluster of
machines to allow data streams
: larger than the capability of any single machine and to
allow clusters of co-ordinated consumers
# systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP
+LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS
+KMOD +IDN
# systemctl cat zookeeper-server.service
# /run/systemd/generator.late/zookeeper-server.service
# Automatically generated by systemd-sysv-generator[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/rc.d/init.d/zookeeper-server
Description=LSB: ZooKeeper is a centralized service for maintaining
configuration information, naming, providing distributed synchronization
Before=runlevel2.target
Before=runlevel3.target
Before=runlevel4.target
Before=runlevel5.target
Before=shutdown.target
After=network-online.target
After=nss-lookup.target
After=network-online.target
Wants=network-online.target
Conflicts=shutdown.target[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=no
PIDFile=/var/run/zookeeper/zookeeper-server.pid
ExecStart=/etc/rc.d/init.d/zookeeper-server start
ExecStop=/etc/rc.d/init.d/zookeeper-server stop
Thanks in advance for your time!
--
Kind regards,
Tasos
Re: systemd failed to stop zookeeper-server.
Posted by Szalay-Bekő Máté <sz...@gmail.com>.
Hi Tasos,
Thanks for reporting the issue!
Taking a quick look on it, I don't think it is related to ZooKeeper itself,
rather it is something related to systemd and to the init scripts / service
descriptors on CentOS. We don't maintain those parts. I actually never used
systemd in production to run ZooKeeper as a service, but use custom
in-house management scripts to start / stop / monitor the ZooKeeper
process. But who knows, maybe someone has some experience with this in the
ZooKeeper community too...
If no one reply you on this list, then maybe someone in the CentOS
community or in BigTop can help you to fix the issue.
Kind regards,
Mate
On Wed, Mar 4, 2020 at 3:44 PM Anastasios Lisgaras
<ta...@yahoo.gr.invalid> wrote:
> Hello zookeeper community,
>
> After the last system update I had a problem with the zookeeper-server.
> The problem was about the same as this:
> https://issues.apache.org/jira/browse/BIGTOP-3302
> So, I went to the file :
> /etc/init.d/zookeeper-server
>
> and I replaced "su" with "runuser". If you want to see it :
>
> # cat /etc/init.d/zookeeper-server : https://termbin.com/gupi
> # cat /usr/lib/zookeeper/bin/zkServer.sh : https://termbin.com/3xpb
>
> After this change the server runs flawlessly, but when i try to stop it
> via "systemctl" the system (systemd) complains.
>
> ```
> # systemctl status zookeeper-server
> ● zookeeper-server.service - LSB: ZooKeeper is a centralized service for
> maintaining configuration information, naming, providing distributed
> synchronization, and providing group services.
> Loaded: loaded (/etc/rc.d/init.d/zookeeper-server; bad; vendor
> preset: disabled)
> Active: active (running) since Wed 2020-03-04 16:16:51 EET; 2s ago
> Docs: man:systemd-sysv-generator(8)
> Process: 30509 ExecStart=/etc/rc.d/init.d/zookeeper-server start
> (code=exited, status=0/SUCCESS)
> Main PID: 30557 (java)
> CGroup: /system.slice/zookeeper-server.service
> └─30557 /usr/lib/jvm/jre-openjdk/bin/java
> -Dzookeeper.datadir.autocreate=false
> -Dzookeeper.log.dir=/var/log/zookeeper -Dzookee...
>
> Mar 04 16:16:50 myserver systemd[1]: Starting LSB: ZooKeeper is a
> centralized service for maintaining configuration information,...ices....
> Mar 04 16:16:50 myserver runuser[30541]: pam_unix(runuser:session):
> session opened for user zookeeper by (uid=0)
> Mar 04 16:16:50 myserver zookeeper-server[30509]: JMX enabled by default
> Mar 04 16:16:50 myserver zookeeper-server[30509]: Using config:
> /etc/zookeeper/conf/zoo.cfg
> Mar 04 16:16:51 myserver zookeeper-server[30509]: Starting zookeeper ...
> STARTED
> Mar 04 16:16:51 myserver runuser[30541]: pam_unix(runuser:session):
> session closed for user zookeeper
> Mar 04 16:16:51 myserver systemd[1]: Started LSB: ZooKeeper is a
> centralized service for maintaining configuration information, ...rvices..
> Hint: Some lines were ellipsized, use -l to show in full.
>
>
> # systemctl stop zookeeper-server
>
>
> # systemctl status zookeeper-server -l
> ● zookeeper-server.service - LSB: ZooKeeper is a centralized service for
> maintaining configuration information, naming, providing distributed
> synchronization, and providing group services.
> Loaded: loaded (/etc/rc.d/init.d/zookeeper-server; bad; vendor
> preset: disabled)
> Active: failed (Result: signal) since Wed 2020-03-04 16:17:00 EET; 4s
> ago
> Docs: man:systemd-sysv-generator(8)
> Process: 30605 ExecStop=/etc/rc.d/init.d/zookeeper-server stop
> (code=exited, status=0/SUCCESS)
> Process: 30509 ExecStart=/etc/rc.d/init.d/zookeeper-server start
> (code=exited, status=0/SUCCESS)
> Main PID: 30557 (code=killed, signal=KILL)
>
> Mar 04 16:16:51 myserver systemd[1]: Started LSB: ZooKeeper is a
> centralized service for maintaining configuration information, naming,
> providing distributed synchronization, and providing group services..
> Mar 04 16:17:00 myserver systemd[1]: Stopping LSB: ZooKeeper is a
> centralized service for maintaining configuration information, naming,
> providing distributed synchronization, and providing group services....
> Mar 04 16:17:00 myserver runuser[30642]: pam_unix(runuser:session):
> session opened for user zookeeper by (uid=0)
> Mar 04 16:17:00 myserver zookeeper-server[30605]: JMX enabled by default
> Mar 04 16:17:00 myserver zookeeper-server[30605]: Using config:
> /etc/zookeeper/conf/zoo.cfg
> Mar 04 16:17:00 myserver zookeeper-server[30605]: Stopping zookeeper ...
> STOPPED
> Mar 04 16:17:00 myserver systemd[1]: zookeeper-server.service: main
> process exited, code=killed, status=9/KILL
> Mar 04 16:17:00 myserver systemd[1]: Stopped LSB: ZooKeeper is a
> centralized service for maintaining configuration information, naming,
> providing distributed synchronization, and providing group services..
> Mar 04 16:17:00 myserver systemd[1]: Unit zookeeper-server.service
> entered failed state.
> Mar 04 16:17:00 myserver systemd[1]: zookeeper-server.service failed.
>
>
> # systemctl is-active zookeeper-server
> failed
>
>
> # systemctl is-failed zookeeper-server
> failed
>
>
> # ps -aux | grep zookeeper
> root 31181 0.0 0.0 112712 968 pts/0 S+ 16:21 0:00 grep
> --color=auto zookeeper
>
>
> # systemctl restart zookeeper-server -l
>
>
> # systemctl status zookeeper-server -l
> ● zookeeper-server.service - LSB: ZooKeeper is a centralized service for
> maintaining configuration information, naming, providing distributed
> synchronization, and providing group services.
> Loaded: loaded (/etc/rc.d/init.d/zookeeper-server; bad; vendor
> preset: disabled)
> Active: active (running) since Wed 2020-03-04 16:18:37 EET; 2s ago
> Docs: man:systemd-sysv-generator(8)
> Process: 30605 ExecStop=/etc/rc.d/init.d/zookeeper-server stop
> (code=exited, status=0/SUCCESS)
> Process: 30746 ExecStart=/etc/rc.d/init.d/zookeeper-server start
> (code=exited, status=0/SUCCESS)
> Main PID: 30794 (java)
> CGroup: /system.slice/zookeeper-server.service
> └─30794 /usr/lib/jvm/jre-openjdk/bin/java
> -Dzookeeper.datadir.autocreate=false
> -Dzookeeper.log.dir=/var/log/zookeeper
> -Dzookeeper.root.logger=INFO,ROLLINGFILE -cp
>
> /usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/zookeeper/bin/../lib/jline-2.11.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.16.2.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/*:/usr/lib/zookeeper/lib/*
> -Dzookeeper.log.threshold=INFO -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.local.only=false
> org.apache.zookeeper.server.quorum.QuorumPeerMain
> /etc/zookeeper/conf/zoo.cfg
>
> Mar 04 16:18:35 myserver systemd[1]: Starting LSB: ZooKeeper is a
> centralized service for maintaining configuration information, naming,
> providing distributed synchronization, and providing group services....
> Mar 04 16:18:36 myserver runuser[30778]: pam_unix(runuser:session):
> session opened for user zookeeper by (uid=0)
> Mar 04 16:18:36 myserver zookeeper-server[30746]: JMX enabled by default
> Mar 04 16:18:36 myserver zookeeper-server[30746]: Using config:
> /etc/zookeeper/conf/zoo.cfg
> Mar 04 16:18:37 myserver zookeeper-server[30746]: Starting zookeeper ...
> STARTED
> Mar 04 16:18:37 myserver systemd[1]: Started LSB: ZooKeeper is a
> centralized service for maintaining configuration information, naming,
> providing distributed synchronization, and providing group services..
>
>
> # ps -aux | grep zookeeper
> zookeep+ 30794 1.0 1.8 3499256 72648 ? Sl 16:18 0:02
> /usr/lib/jvm/jre-openjdk/bin/java -Dzookeeper.datadir.autocreate=false
> -Dzookeeper.log.dir=/var/log/zookeeper
> -Dzookeeper.root.logger=INFO,ROLLINGFILE -cp
>
> /usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/zookeeper/bin/../lib/jline-2.11.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.16.2.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/*:/usr/lib/zookeeper/lib/*
> -Dzookeeper.log.threshold=INFO -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.local.only=false
> org.apache.zookeeper.server.quorum.QuorumPeerMain
> /etc/zookeeper/conf/zoo.cfg
> root 31105 0.0 0.0 112712 968 pts/0 R+ 16:21 0:00 grep
> --color=auto zookeeper
> ```
>
> As you can see, starting, and restarting works properly.
> When I stop the zookeeper-server, then the systemctl in the status
> indicates a problem, but the zookeeper-server actually stops.
>
> Can you please help me to fix this problem ?
> Because this problem (about systemd ) is relatively recent, the
> development team is aware of it ?
>
> OS : CentOS Linux release 7.7.1908 (Core)
>
> # yum info zookeeper-server
> Loaded plugins: fastestmirror
> Loading mirror speeds from cached hostfile
> * base: ftp.ntua.gr
> * epel: mirrors.daticum.com
> * extras: ftp.ntua.gr
> * updates: ftp.ntua.gr
> Installed Packages
> Name : zookeeper-server
> Arch : x86_64
> Version : 3.4.5+cdh5.16.2+159
> Release : 1.cdh5.16.2.p0.23.el7
> Size : 5.0 k
> Repo : installed
> From repo : cloudera-cdh5
> Summary : The Hadoop Zookeeper server
> URL : http://zookeeper.apache.org/
> License : ASL 2.0
> Description : This package starts the zookeeper server on startup
>
>
> # yum info kafka
> Loaded plugins: fastestmirror
> Loading mirror speeds from cached hostfile
> * base: ftp.ntua.gr
> * epel: mirror.hosting90.cz
> * extras: ftp.ntua.gr
> * updates: ftp.ntua.gr
> Installed Packages
> Name : kafka
> Arch : noarch
> Version : 2.2.1+kafka4.1.0
> Release : 2.4.1.0.p0.6.el7
> Size : 91 M
> Repo : installed
> From repo : cloudera-kafka
> Summary : Apache Kafka is publish-subscribe messaging rethought as a
> distributed commit log.
> URL : http://kafka.apache.org/
> License : ASL 2.0
> Description : Apache Kafka is publish-subscribe messaging rethought as a
> distributed commit log.
> : A single Kafka broker can handle hundreds of megabytes of
> reads and writes per second
> : from thousands of clients. It can be elastically and
> transparently expanded without downtime.
> : Data streams are partitioned and spread over a cluster of
> machines to allow data streams
> : larger than the capability of any single machine and to
> allow clusters of co-ordinated consumers
>
>
> # systemctl --version
> systemd 219
> +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP
> +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS
> +KMOD +IDN
>
>
> # systemctl cat zookeeper-server.service
> # /run/systemd/generator.late/zookeeper-server.service
> # Automatically generated by systemd-sysv-generator[Unit]
> Documentation=man:systemd-sysv-generator(8)
> SourcePath=/etc/rc.d/init.d/zookeeper-server
> Description=LSB: ZooKeeper is a centralized service for maintaining
> configuration information, naming, providing distributed synchronization
> Before=runlevel2.target
> Before=runlevel3.target
> Before=runlevel4.target
> Before=runlevel5.target
> Before=shutdown.target
> After=network-online.target
> After=nss-lookup.target
> After=network-online.target
> Wants=network-online.target
> Conflicts=shutdown.target[Service]
> Type=forking
> Restart=no
> TimeoutSec=5min
> IgnoreSIGPIPE=no
> KillMode=process
> GuessMainPID=no
> RemainAfterExit=no
> PIDFile=/var/run/zookeeper/zookeeper-server.pid
> ExecStart=/etc/rc.d/init.d/zookeeper-server start
> ExecStop=/etc/rc.d/init.d/zookeeper-server stop
>
>
> Thanks in advance for your time!
>
> --
> Kind regards,
> Tasos
>
>
Re: systemd failed to stop zookeeper-server.
Posted by Andor Molnar <an...@apache.org>.
Hi,
Strange. It’s actually normal that ZooKeeper process gets stopped with the kill signal. That’s "business as usual”, but why does systemd have problems with that and since when has this been happening?
Andor
> On 2020. Mar 4., at 15:44, Anastasios Lisgaras <ta...@yahoo.gr.INVALID> wrote:
>
> Hello zookeeper community,
>
> After the last system update I had a problem with the zookeeper-server.
> The problem was about the same as this:
> https://issues.apache.org/jira/browse/BIGTOP-3302
> So, I went to the file :
> /etc/init.d/zookeeper-server
>
> and I replaced "su" with "runuser". If you want to see it :
>
> # cat /etc/init.d/zookeeper-server : https://termbin.com/gupi
> # cat /usr/lib/zookeeper/bin/zkServer.sh : https://termbin.com/3xpb
>
> After this change the server runs flawlessly, but when i try to stop it
> via "systemctl" the system (systemd) complains.
>
> ```
> # systemctl status zookeeper-server
> ● zookeeper-server.service - LSB: ZooKeeper is a centralized service for
> maintaining configuration information, naming, providing distributed
> synchronization, and providing group services.
> Loaded: loaded (/etc/rc.d/init.d/zookeeper-server; bad; vendor
> preset: disabled)
> Active: active (running) since Wed 2020-03-04 16:16:51 EET; 2s ago
> Docs: man:systemd-sysv-generator(8)
> Process: 30509 ExecStart=/etc/rc.d/init.d/zookeeper-server start
> (code=exited, status=0/SUCCESS)
> Main PID: 30557 (java)
> CGroup: /system.slice/zookeeper-server.service
> └─30557 /usr/lib/jvm/jre-openjdk/bin/java
> -Dzookeeper.datadir.autocreate=false
> -Dzookeeper.log.dir=/var/log/zookeeper -Dzookee...
>
> Mar 04 16:16:50 myserver systemd[1]: Starting LSB: ZooKeeper is a
> centralized service for maintaining configuration information,...ices....
> Mar 04 16:16:50 myserver runuser[30541]: pam_unix(runuser:session):
> session opened for user zookeeper by (uid=0)
> Mar 04 16:16:50 myserver zookeeper-server[30509]: JMX enabled by default
> Mar 04 16:16:50 myserver zookeeper-server[30509]: Using config:
> /etc/zookeeper/conf/zoo.cfg
> Mar 04 16:16:51 myserver zookeeper-server[30509]: Starting zookeeper ...
> STARTED
> Mar 04 16:16:51 myserver runuser[30541]: pam_unix(runuser:session):
> session closed for user zookeeper
> Mar 04 16:16:51 myserver systemd[1]: Started LSB: ZooKeeper is a
> centralized service for maintaining configuration information, ...rvices..
> Hint: Some lines were ellipsized, use -l to show in full.
>
>
> # systemctl stop zookeeper-server
>
>
> # systemctl status zookeeper-server -l
> ● zookeeper-server.service - LSB: ZooKeeper is a centralized service for
> maintaining configuration information, naming, providing distributed
> synchronization, and providing group services.
> Loaded: loaded (/etc/rc.d/init.d/zookeeper-server; bad; vendor
> preset: disabled)
> Active: failed (Result: signal) since Wed 2020-03-04 16:17:00 EET; 4s ago
> Docs: man:systemd-sysv-generator(8)
> Process: 30605 ExecStop=/etc/rc.d/init.d/zookeeper-server stop
> (code=exited, status=0/SUCCESS)
> Process: 30509 ExecStart=/etc/rc.d/init.d/zookeeper-server start
> (code=exited, status=0/SUCCESS)
> Main PID: 30557 (code=killed, signal=KILL)
>
> Mar 04 16:16:51 myserver systemd[1]: Started LSB: ZooKeeper is a
> centralized service for maintaining configuration information, naming,
> providing distributed synchronization, and providing group services..
> Mar 04 16:17:00 myserver systemd[1]: Stopping LSB: ZooKeeper is a
> centralized service for maintaining configuration information, naming,
> providing distributed synchronization, and providing group services....
> Mar 04 16:17:00 myserver runuser[30642]: pam_unix(runuser:session):
> session opened for user zookeeper by (uid=0)
> Mar 04 16:17:00 myserver zookeeper-server[30605]: JMX enabled by default
> Mar 04 16:17:00 myserver zookeeper-server[30605]: Using config:
> /etc/zookeeper/conf/zoo.cfg
> Mar 04 16:17:00 myserver zookeeper-server[30605]: Stopping zookeeper ...
> STOPPED
> Mar 04 16:17:00 myserver systemd[1]: zookeeper-server.service: main
> process exited, code=killed, status=9/KILL
> Mar 04 16:17:00 myserver systemd[1]: Stopped LSB: ZooKeeper is a
> centralized service for maintaining configuration information, naming,
> providing distributed synchronization, and providing group services..
> Mar 04 16:17:00 myserver systemd[1]: Unit zookeeper-server.service
> entered failed state.
> Mar 04 16:17:00 myserver systemd[1]: zookeeper-server.service failed.
>
>
> # systemctl is-active zookeeper-server
> failed
>
>
> # systemctl is-failed zookeeper-server
> failed
>
>
> # ps -aux | grep zookeeper
> root 31181 0.0 0.0 112712 968 pts/0 S+ 16:21 0:00 grep
> --color=auto zookeeper
>
>
> # systemctl restart zookeeper-server -l
>
>
> # systemctl status zookeeper-server -l
> ● zookeeper-server.service - LSB: ZooKeeper is a centralized service for
> maintaining configuration information, naming, providing distributed
> synchronization, and providing group services.
> Loaded: loaded (/etc/rc.d/init.d/zookeeper-server; bad; vendor
> preset: disabled)
> Active: active (running) since Wed 2020-03-04 16:18:37 EET; 2s ago
> Docs: man:systemd-sysv-generator(8)
> Process: 30605 ExecStop=/etc/rc.d/init.d/zookeeper-server stop
> (code=exited, status=0/SUCCESS)
> Process: 30746 ExecStart=/etc/rc.d/init.d/zookeeper-server start
> (code=exited, status=0/SUCCESS)
> Main PID: 30794 (java)
> CGroup: /system.slice/zookeeper-server.service
> └─30794 /usr/lib/jvm/jre-openjdk/bin/java
> -Dzookeeper.datadir.autocreate=false
> -Dzookeeper.log.dir=/var/log/zookeeper
> -Dzookeeper.root.logger=INFO,ROLLINGFILE -cp
> /usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/zookeeper/bin/../lib/jline-2.11.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.16.2.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/*:/usr/lib/zookeeper/lib/*
> -Dzookeeper.log.threshold=INFO -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.local.only=false
> org.apache.zookeeper.server.quorum.QuorumPeerMain
> /etc/zookeeper/conf/zoo.cfg
>
> Mar 04 16:18:35 myserver systemd[1]: Starting LSB: ZooKeeper is a
> centralized service for maintaining configuration information, naming,
> providing distributed synchronization, and providing group services....
> Mar 04 16:18:36 myserver runuser[30778]: pam_unix(runuser:session):
> session opened for user zookeeper by (uid=0)
> Mar 04 16:18:36 myserver zookeeper-server[30746]: JMX enabled by default
> Mar 04 16:18:36 myserver zookeeper-server[30746]: Using config:
> /etc/zookeeper/conf/zoo.cfg
> Mar 04 16:18:37 myserver zookeeper-server[30746]: Starting zookeeper ...
> STARTED
> Mar 04 16:18:37 myserver systemd[1]: Started LSB: ZooKeeper is a
> centralized service for maintaining configuration information, naming,
> providing distributed synchronization, and providing group services..
>
>
> # ps -aux | grep zookeeper
> zookeep+ 30794 1.0 1.8 3499256 72648 ? Sl 16:18 0:02
> /usr/lib/jvm/jre-openjdk/bin/java -Dzookeeper.datadir.autocreate=false
> -Dzookeeper.log.dir=/var/log/zookeeper
> -Dzookeeper.root.logger=INFO,ROLLINGFILE -cp
> /usr/lib/zookeeper/bin/../build/classes:/usr/lib/zookeeper/bin/../build/lib/*.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12.jar:/usr/lib/zookeeper/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/slf4j-api-1.7.5.jar:/usr/lib/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/usr/lib/zookeeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/zookeeper/bin/../lib/jline-2.11.jar:/usr/lib/zookeeper/bin/../zookeeper-3.4.5-cdh5.16.2.jar:/usr/lib/zookeeper/bin/../src/java/lib/*.jar:/etc/zookeeper/conf::/etc/zookeeper/conf:/usr/lib/zookeeper/*:/usr/lib/zookeeper/lib/*
> -Dzookeeper.log.threshold=INFO -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.local.only=false
> org.apache.zookeeper.server.quorum.QuorumPeerMain
> /etc/zookeeper/conf/zoo.cfg
> root 31105 0.0 0.0 112712 968 pts/0 R+ 16:21 0:00 grep
> --color=auto zookeeper
> ```
>
> As you can see, starting, and restarting works properly.
> When I stop the zookeeper-server, then the systemctl in the status
> indicates a problem, but the zookeeper-server actually stops.
>
> Can you please help me to fix this problem ?
> Because this problem (about systemd ) is relatively recent, the
> development team is aware of it ?
>
> OS : CentOS Linux release 7.7.1908 (Core)
>
> # yum info zookeeper-server
> Loaded plugins: fastestmirror
> Loading mirror speeds from cached hostfile
> * base: ftp.ntua.gr
> * epel: mirrors.daticum.com
> * extras: ftp.ntua.gr
> * updates: ftp.ntua.gr
> Installed Packages
> Name : zookeeper-server
> Arch : x86_64
> Version : 3.4.5+cdh5.16.2+159
> Release : 1.cdh5.16.2.p0.23.el7
> Size : 5.0 k
> Repo : installed
> From repo : cloudera-cdh5
> Summary : The Hadoop Zookeeper server
> URL : http://zookeeper.apache.org/
> License : ASL 2.0
> Description : This package starts the zookeeper server on startup
>
>
> # yum info kafka
> Loaded plugins: fastestmirror
> Loading mirror speeds from cached hostfile
> * base: ftp.ntua.gr
> * epel: mirror.hosting90.cz
> * extras: ftp.ntua.gr
> * updates: ftp.ntua.gr
> Installed Packages
> Name : kafka
> Arch : noarch
> Version : 2.2.1+kafka4.1.0
> Release : 2.4.1.0.p0.6.el7
> Size : 91 M
> Repo : installed
> From repo : cloudera-kafka
> Summary : Apache Kafka is publish-subscribe messaging rethought as a
> distributed commit log.
> URL : http://kafka.apache.org/
> License : ASL 2.0
> Description : Apache Kafka is publish-subscribe messaging rethought as a
> distributed commit log.
> : A single Kafka broker can handle hundreds of megabytes of
> reads and writes per second
> : from thousands of clients. It can be elastically and
> transparently expanded without downtime.
> : Data streams are partitioned and spread over a cluster of
> machines to allow data streams
> : larger than the capability of any single machine and to
> allow clusters of co-ordinated consumers
>
>
> # systemctl --version
> systemd 219
> +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP
> +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS
> +KMOD +IDN
>
>
> # systemctl cat zookeeper-server.service
> # /run/systemd/generator.late/zookeeper-server.service
> # Automatically generated by systemd-sysv-generator[Unit]
> Documentation=man:systemd-sysv-generator(8)
> SourcePath=/etc/rc.d/init.d/zookeeper-server
> Description=LSB: ZooKeeper is a centralized service for maintaining
> configuration information, naming, providing distributed synchronization
> Before=runlevel2.target
> Before=runlevel3.target
> Before=runlevel4.target
> Before=runlevel5.target
> Before=shutdown.target
> After=network-online.target
> After=nss-lookup.target
> After=network-online.target
> Wants=network-online.target
> Conflicts=shutdown.target[Service]
> Type=forking
> Restart=no
> TimeoutSec=5min
> IgnoreSIGPIPE=no
> KillMode=process
> GuessMainPID=no
> RemainAfterExit=no
> PIDFile=/var/run/zookeeper/zookeeper-server.pid
> ExecStart=/etc/rc.d/init.d/zookeeper-server start
> ExecStop=/etc/rc.d/init.d/zookeeper-server stop
>
>
> Thanks in advance for your time!
>
> --
> Kind regards,
> Tasos
>