zkCli.sh - JMX args conflict

[Blaise Frédéric <fr...@kudelskisecurity.com>]

Hello all,

I am fairly new to ZK, and have deployed my first cluster (via puppet). Everything looks good except for one little glitch when wanting to connect with zkCli.sh with specific JMX params set.

Background info
=============
# rpm -qa |grep zook
zookeeper-3.4.5+cdh5.8.0+94-1.cdh5.8.0.p0.76.el7.x86_64
zookeeper-server-3.4.5+cdh5.8.0+94-1.cdh5.8.0.p0.76.el7.x86_64

# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

The problem
==========
I cannot connect using '/usr/lib/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181’. I am using JVMFLAGS to setup specific options.

I can connect with jconsole remotely just fine or any other JMX monitoring utilities. I can also connect on localhost, the server ip or dns name from the machine itself.
There is no iptables. No selinux.

My starting point is, as exported in my java.env file under /etc/zookeeper/conf :

export JVMFLAGS="-Xms512m -Xmx1g -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.rmi.port=12345 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.password.file=/etc/java/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/etc/java/jmxremote.access -Djava.rmi.server.hostname=${HOSTIP} -Dcom.sun.management.jmxremote.authenticate=true -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -XX:+PrintGCCause -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/zookeeper/gc-zk.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=256M -XX:+HeapDumpOnOutOfMemoryError -XX:OnOutOfMemoryError='/usr/local/bin/kill_zk_jvm.sh'"

1. If using either one of “java.rmi.server.hostname” and “comment.sun.management.jmxremote.rmi.port”, then it fails with:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
# /usr/lib/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181
Error: JMX connector server communication error: service:jmx:rmi://i-infra-zk-001.my.domain:12345

2. If removing the rmi parameters, then it fails with:
--------------------------------------------------------------------
# /usr/lib/zookeeper/bin/zkCli.sh -server 127.0.0.1:2181
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 12345; nested exception is:
java.net.BindException: Address already in use

If I don’t have any JMX parameters in there, it works fine.

Thank you for your advices or pointers. I could not find much that pointed at this particular issue; I guess I am missing something fairly obvious...

Cheers,
fred