You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Mark Johnson <ma...@servicemesh.com> on 2017/09/19 15:16:06 UTC
Problem creating Ephemeral znodes
Hi,
I'm running version 3.5.3-BETA of Zookeeper in a Docker container and
usually cannot create Ephemeral znodes, either through the CLI or though
a Java program. It appears the owner is not set. Here's an example
session:
============================================================================================================
[root@Appliance ~]# docker exec -it 9b7cd1e80b5b /bin/bash
zookeeper@9b7cd1e80b5b:/opt/zookeeper-3.5.3-beta$ bin/zkCli.sh
Connecting to localhost:2181
2017-09-19 10:03:31,373 [myid:] - INFO [main:Environment@109] - Client
environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60,
built on 04/03/2017 16:19 GMT
2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client
environment:host.name=9b7cd1e80b5b
2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client
environment:java.version=1.8.0_144
2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
environment:java.vendor=Oracle Corporation
2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
environment:java.home=/usr/lib/jvm/java-8-oracle/jre
2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
environment:java.class.path=/opt/zookeeper-3.5.3-beta/bin/../build/classes:/opt/zookeeper-3.5.3-beta/bin/../build/lib/*.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar
:/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jline-2.11.jar:/opt/zookeeper-3.5.3-beta/bin/..
/lib/jetty-util-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-server-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-security-9.2.18.v20160721.jar:/op
t/zookeeper-3.5.3-beta/bin/../lib/jetty-io-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-http-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jackson-mapper-asl-1.
9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jackson-core-asl-1.9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper-3.5.3-beta/bin/../zookeeper-3.5.3-beta.jar:/opt/zookeeper-3.5.3-beta/bin/../src/java/lib/*.jar:/opt/zookeeper-
3.5.3-beta/bin/../conf:
2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
environment:java.io.tmpdir=/tmp
2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
environment:java.compiler=<NA>
2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
environment:os.name=Linux
2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
environment:os.arch=amd64
2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
environment:os.version=3.10.0-514.26.2.el7.x86_64
2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
environment:user.name=zookeeper
2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
environment:user.home=/home/zookeeper
2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
environment:user.dir=/opt/zookeeper-3.5.3-beta
2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
environment:os.memory.free=233MB
2017-09-19 10:03:31,382 [myid:] - INFO [main:Environment@109] - Client
environment:os.memory.max=240MB
2017-09-19 10:03:31,383 [myid:] - INFO [main:Environment@109] - Client
environment:os.memory.total=240MB
2017-09-19 10:03:31,386 [myid:] - INFO [main:ZooKeeper@865] -
Initiating client connection, connectString=localhost:2181
sessionTimeout=30000
watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@28ba21f3
2017-09-19 10:03:31,397 [myid:] - INFO [main:ClientCnxnSocket@236] -
jute.maxbuffer value is 4194304 Bytes
2017-09-19 10:03:31,410 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1113] - Opening
socket connection to server localhost/127.0.0.1:2181. Will not attempt
to authenticate using SASL (unknown error)
Welcome to ZooKeeper!
JLine support is enabled
2017-09-19 10:03:31,493 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@948] - Socket
connection established, initiating session, client: /127.0.0.1:44882,
server: localhost/127.0.0.1:2181
2017-09-19 10:03:31,506 [myid:localhost:2181] - INFO
[main-SendThread(localhost:2181):ClientCnxn$SendThread@1381] - Session
establishment complete on server localhost/127.0.0.1:2181, sessionid =
0xeb0000480b3c005c, negotiated timeout = 30000
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] create -e /testnode
Created /testnode
[zk: localhost:2181(CONNECTED) 1] stat /testnode
cZxid = 0x2000002c4
ctime = Tue Sep 19 10:03:50 CDT 2017
mZxid = 0x2000002c4
mtime = Tue Sep 19 10:03:50 CDT 2017
pZxid = 0x2000002c4
cversion = 0
dataVersion = 0
aclVersion = 0
*ephemeralOwner = 0x0*
dataLength = 0
numChildren = 0
[zk: localhost:2181(CONNECTED) 2]
==========================================================================================================
If I disconnect and reconnect, the znode is still present as well. If I
restart Zookeeper, many times this will resolve itself somehow and
creation of ephemeral nodes will work correctly. I have also tried this
in a clustered configuration and get cases where one Zookeeper will
create ephemeral nodes correctly and another Zookeeper will exhibit the
above issue.
Please let me know what other information would be of use or other
things I could be looking for.
Any help appreciated.
Mark
Re: Problem creating Ephemeral znodes
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
I didn't think client ID could be negative. Please open an issue in Jira and assign this to me.
-Jordan
> On Sep 20, 2017, at 1:09 PM, Mark Johnson <ma...@servicemesh.com> wrote:
>
> Think I found the problem. In the code that determines the EphemeralType
> it is looking at the owner (which is the client ID or connection ID):
>
> EphemeralType.java:
>
> public static EphemeralType get(long ephemeralOwner) {
> if (ephemeralOwner == CONTAINER_EPHEMERAL_OWNER) {
> return CONTAINER;
> }
> if (ephemeralOwner < 0) {
> return TTL;
> }
> return (ephemeralOwner == 0) ? VOID : NORMAL;
> }
>
> However my connection ID is:
>
> header.getClientId(): -720548323429908480
>
> This causes the code to think this is a TTL Ephemeral node instead of a
> NORMAL Ephemeral node.
>
> This also explains why this is random - if my client ID is non-negative
> then the node gets added correctly.
>
> Can someone verify that this is a bug? Can a ticket be created for this?
>
> Thanks in advance,
> Mark
>
>
> On Wed, Sep 20, 2017 at 1:17 AM, Mark Johnson <ma...@servicemesh.com>
> wrote:
>
>> Hi Abraham,
>>
>> I believe I'm using nio - from the log:
>>
>> 2017-09-19 15:25:51,761 [myid:010000001020] - INFO
>> [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with
>> 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads,
>> and 64 kB direct buffers.
>>
>> And here's my zoo.cfg:
>>
>> tickTime=2000
>> initLimit=100
>> syncLimit=5
>> dataDir=/var/lib/zookeeper
>> clientPort=2181
>> autopurge.snapRetainCount=3
>> # Purge task interval in hours
>> autopurge.purgeInterval=1
>> minSessionTimeout=4000
>> maxSessionTimeout=60000
>> standaloneEnabled=false
>> reconfigEnabled=true
>> skipACL=yes
>> dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic
>>
>> *However, the problem is not removal of the ephemeral nodes, it's that
>> they're not getting created in the first place,* When I use the CLI to
>> create an ephemeral node via:
>>
>> create -e /testnode
>>
>> It's not marked as an ephemeral node and hangs around forever. From stat
>> /testnode:
>>
>> ephemeralOwner = 0x0
>>
>> Sometimes Zookeeper will come up and create ephemeral nodes and they work
>> correctly (deleted after the connection is terminated). But usually (with
>> the version I'm using- 3.5.3-beta2) they are not created.
>>
>> Are you familiar with the code? I've downloaded the source and have been
>> looking for the place where the znodes are actually created but have not
>> found that section. If you can point me to the right area of the code I
>> can probably troubleshoot this issue.
>>
>> Thanks for any light you can shed on this problem!
>>
>> Mark
>>
>>
>> On Tue, Sep 19, 2017 at 6:18 PM, Abraham Fine <af...@apache.org> wrote:
>>
>>> Hi Mark-
>>>
>>> I'm assuming you are using nio and not netty, please let me know if that
>>> is not the case.
>>>
>>> Have you possibly set zookeeper.nio.sessionlessCnxnTimeout? This value
>>> is 10s by default, so it may take a little more than ten seconds by
>>> default for ephemeral nodes to be deleted.
>>>
>>> Thanks,
>>> Abe
>>>
>>>
>>> On Tue, Sep 19, 2017, at 08:16, Mark Johnson wrote:
>>>> Hi,
>>>>
>>>> I'm running version 3.5.3-BETA of Zookeeper in a Docker container and
>>>> usually cannot create Ephemeral znodes, either through the CLI or though
>>>> a Java program. It appears the owner is not set. Here's an example
>>>> session:
>>>>
>>>> ============================================================
>>> ================================================
>>>> [root@Appliance ~]# docker exec -it 9b7cd1e80b5b /bin/bash
>>>> zookeeper@9b7cd1e80b5b:/opt/zookeeper-3.5.3-beta$ bin/zkCli.sh
>>>> Connecting to localhost:2181
>>>> 2017-09-19 10:03:31,373 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8
>>> f21a47e06b42864475a60,
>>>> built on 04/03/2017 16:19 GMT
>>>> 2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:host.name=9b7cd1e80b5b
>>>> 2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:java.version=1.8.0_144
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:java.vendor=Oracle Corporation
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:java.class.path=/opt/zookeeper-3.5.3-beta/bin/..
>>> /build/classes:/opt/zookeeper-3.5.3-beta/bin/../build/lib/*.
>>> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar
>>>> :/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.jar:/
>>> opt/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Final.jar:/
>>> opt/zookeeper-3.5.3-beta/bin/../lib/log4j-1.2.17.jar:/opt/
>>> zookeeper-3.5.3-beta/bin/../lib/jline-2.11.jar:/opt/zookee
>>> per-3.5.3-beta/bin/..
>>>> /lib/jetty-util-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-
>>> beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/opt/
>>> zookeeper-3.5.3-beta/bin/../lib/jetty-server-9.2.18.v20160
>>> 721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-security
>>> -9.2.18.v20160721.jar:/op
>>>> t/zookeeper-3.5.3-beta/bin/../lib/jetty-io-9.2.18.v20160721.
>>> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-http-9.2.18.v
>>> 20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/javax.serv
>>> let-api-3.1.0.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/
>>> jackson-mapper-asl-1.
>>>> 9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jackson-core-a
>>> sl-1.9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/commons-c
>>> li-1.2.jar:/opt/zookeeper-3.5.3-beta/bin/../zookeeper-3.5.3-
>>> beta.jar:/opt/zookeeper-3.5.3-beta/bin/../src/java/lib/*.jar
>>> :/opt/zookeeper-
>>>> 3.5.3-beta/bin/../conf:
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/
>>> usr/lib64:/lib64:/lib:/usr/lib
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:java.io.tmpdir=/tmp
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:java.compiler=<NA>
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:os.name=Linux
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:os.arch=amd64
>>>> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:os.version=3.10.0-514.26.2.el7.x86_64
>>>> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:user.name=zookeeper
>>>> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:user.home=/home/zookeeper
>>>> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:user.dir=/opt/zookeeper-3.5.3-beta
>>>> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:os.memory.free=233MB
>>>> 2017-09-19 10:03:31,382 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:os.memory.max=240MB
>>>> 2017-09-19 10:03:31,383 [myid:] - INFO [main:Environment@109] - Client
>>>> environment:os.memory.total=240MB
>>>> 2017-09-19 10:03:31,386 [myid:] - INFO [main:ZooKeeper@865] -
>>>> Initiating client connection, connectString=localhost:2181
>>>> sessionTimeout=30000
>>>> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@28ba21f3
>>>> 2017-09-19 10:03:31,397 [myid:] - INFO [main:ClientCnxnSocket@236] -
>>>> jute.maxbuffer value is 4194304 Bytes
>>>> 2017-09-19 10:03:31,410 [myid:localhost:2181] - INFO
>>>> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1113] - Opening
>>>> socket connection to server localhost/127.0.0.1:2181. Will not attempt
>>>> to authenticate using SASL (unknown error)
>>>> Welcome to ZooKeeper!
>>>> JLine support is enabled
>>>> 2017-09-19 10:03:31,493 [myid:localhost:2181] - INFO
>>>> [main-SendThread(localhost:2181):ClientCnxn$SendThread@948] - Socket
>>>> connection established, initiating session, client: /127.0.0.1:44882,
>>>> server: localhost/127.0.0.1:2181
>>>> 2017-09-19 10:03:31,506 [myid:localhost:2181] - INFO
>>>> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1381] - Session
>>>> establishment complete on server localhost/127.0.0.1:2181, sessionid =
>>>> 0xeb0000480b3c005c, negotiated timeout = 30000
>>>>
>>>> WATCHER::
>>>>
>>>> WatchedEvent state:SyncConnected type:None path:null
>>>> [zk: localhost:2181(CONNECTED) 0] create -e /testnode
>>>> Created /testnode
>>>> [zk: localhost:2181(CONNECTED) 1] stat /testnode
>>>> cZxid = 0x2000002c4
>>>> ctime = Tue Sep 19 10:03:50 CDT 2017
>>>> mZxid = 0x2000002c4
>>>> mtime = Tue Sep 19 10:03:50 CDT 2017
>>>> pZxid = 0x2000002c4
>>>> cversion = 0
>>>> dataVersion = 0
>>>> aclVersion = 0
>>>> *ephemeralOwner = 0x0*
>>>> dataLength = 0
>>>> numChildren = 0
>>>> [zk: localhost:2181(CONNECTED) 2]
>>>> ============================================================
>>> ==============================================
>>>>
>>>> If I disconnect and reconnect, the znode is still present as well. If I
>>>> restart Zookeeper, many times this will resolve itself somehow and
>>>> creation of ephemeral nodes will work correctly. I have also tried this
>>>> in a clustered configuration and get cases where one Zookeeper will
>>>> create ephemeral nodes correctly and another Zookeeper will exhibit the
>>>> above issue.
>>>>
>>>> Please let me know what other information would be of use or other
>>>> things I could be looking for.
>>>>
>>>> Any help appreciated.
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>
>>
>>
Re: Problem creating Ephemeral znodes
Posted by Mark Johnson <ma...@servicemesh.com>.
Think I found the problem. In the code that determines the EphemeralType
it is looking at the owner (which is the client ID or connection ID):
EphemeralType.java:
public static EphemeralType get(long ephemeralOwner) {
if (ephemeralOwner == CONTAINER_EPHEMERAL_OWNER) {
return CONTAINER;
}
if (ephemeralOwner < 0) {
return TTL;
}
return (ephemeralOwner == 0) ? VOID : NORMAL;
}
However my connection ID is:
header.getClientId(): -720548323429908480
This causes the code to think this is a TTL Ephemeral node instead of a
NORMAL Ephemeral node.
This also explains why this is random - if my client ID is non-negative
then the node gets added correctly.
Can someone verify that this is a bug? Can a ticket be created for this?
Thanks in advance,
Mark
On Wed, Sep 20, 2017 at 1:17 AM, Mark Johnson <ma...@servicemesh.com>
wrote:
> Hi Abraham,
>
> I believe I'm using nio - from the log:
>
> 2017-09-19 15:25:51,761 [myid:010000001020] - INFO
> [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with
> 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads,
> and 64 kB direct buffers.
>
> And here's my zoo.cfg:
>
> tickTime=2000
> initLimit=100
> syncLimit=5
> dataDir=/var/lib/zookeeper
> clientPort=2181
> autopurge.snapRetainCount=3
> # Purge task interval in hours
> autopurge.purgeInterval=1
> minSessionTimeout=4000
> maxSessionTimeout=60000
> standaloneEnabled=false
> reconfigEnabled=true
> skipACL=yes
> dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic
>
> *However, the problem is not removal of the ephemeral nodes, it's that
> they're not getting created in the first place,* When I use the CLI to
> create an ephemeral node via:
>
> create -e /testnode
>
> It's not marked as an ephemeral node and hangs around forever. From stat
> /testnode:
>
> ephemeralOwner = 0x0
>
> Sometimes Zookeeper will come up and create ephemeral nodes and they work
> correctly (deleted after the connection is terminated). But usually (with
> the version I'm using- 3.5.3-beta2) they are not created.
>
> Are you familiar with the code? I've downloaded the source and have been
> looking for the place where the znodes are actually created but have not
> found that section. If you can point me to the right area of the code I
> can probably troubleshoot this issue.
>
> Thanks for any light you can shed on this problem!
>
> Mark
>
>
> On Tue, Sep 19, 2017 at 6:18 PM, Abraham Fine <af...@apache.org> wrote:
>
>> Hi Mark-
>>
>> I'm assuming you are using nio and not netty, please let me know if that
>> is not the case.
>>
>> Have you possibly set zookeeper.nio.sessionlessCnxnTimeout? This value
>> is 10s by default, so it may take a little more than ten seconds by
>> default for ephemeral nodes to be deleted.
>>
>> Thanks,
>> Abe
>>
>>
>> On Tue, Sep 19, 2017, at 08:16, Mark Johnson wrote:
>> > Hi,
>> >
>> > I'm running version 3.5.3-BETA of Zookeeper in a Docker container and
>> > usually cannot create Ephemeral znodes, either through the CLI or though
>> > a Java program. It appears the owner is not set. Here's an example
>> > session:
>> >
>> > ============================================================
>> ================================================
>> > [root@Appliance ~]# docker exec -it 9b7cd1e80b5b /bin/bash
>> > zookeeper@9b7cd1e80b5b:/opt/zookeeper-3.5.3-beta$ bin/zkCli.sh
>> > Connecting to localhost:2181
>> > 2017-09-19 10:03:31,373 [myid:] - INFO [main:Environment@109] - Client
>> > environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8
>> f21a47e06b42864475a60,
>> > built on 04/03/2017 16:19 GMT
>> > 2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client
>> > environment:host.name=9b7cd1e80b5b
>> > 2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client
>> > environment:java.version=1.8.0_144
>> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>> > environment:java.vendor=Oracle Corporation
>> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>> > environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>> > environment:java.class.path=/opt/zookeeper-3.5.3-beta/bin/..
>> /build/classes:/opt/zookeeper-3.5.3-beta/bin/../build/lib/*.
>> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar
>> > :/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.jar:/
>> opt/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Final.jar:/
>> opt/zookeeper-3.5.3-beta/bin/../lib/log4j-1.2.17.jar:/opt/
>> zookeeper-3.5.3-beta/bin/../lib/jline-2.11.jar:/opt/zookee
>> per-3.5.3-beta/bin/..
>> > /lib/jetty-util-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-
>> beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/opt/
>> zookeeper-3.5.3-beta/bin/../lib/jetty-server-9.2.18.v20160
>> 721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-security
>> -9.2.18.v20160721.jar:/op
>> > t/zookeeper-3.5.3-beta/bin/../lib/jetty-io-9.2.18.v20160721.
>> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-http-9.2.18.v
>> 20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/javax.serv
>> let-api-3.1.0.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/
>> jackson-mapper-asl-1.
>> > 9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jackson-core-a
>> sl-1.9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/commons-c
>> li-1.2.jar:/opt/zookeeper-3.5.3-beta/bin/../zookeeper-3.5.3-
>> beta.jar:/opt/zookeeper-3.5.3-beta/bin/../src/java/lib/*.jar
>> :/opt/zookeeper-
>> > 3.5.3-beta/bin/../conf:
>> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>> > environment:java.library.path=/usr/java/packages/lib/amd64:/
>> usr/lib64:/lib64:/lib:/usr/lib
>> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>> > environment:java.io.tmpdir=/tmp
>> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>> > environment:java.compiler=<NA>
>> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>> > environment:os.name=Linux
>> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
>> > environment:os.arch=amd64
>> > 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
>> > environment:os.version=3.10.0-514.26.2.el7.x86_64
>> > 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
>> > environment:user.name=zookeeper
>> > 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
>> > environment:user.home=/home/zookeeper
>> > 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
>> > environment:user.dir=/opt/zookeeper-3.5.3-beta
>> > 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
>> > environment:os.memory.free=233MB
>> > 2017-09-19 10:03:31,382 [myid:] - INFO [main:Environment@109] - Client
>> > environment:os.memory.max=240MB
>> > 2017-09-19 10:03:31,383 [myid:] - INFO [main:Environment@109] - Client
>> > environment:os.memory.total=240MB
>> > 2017-09-19 10:03:31,386 [myid:] - INFO [main:ZooKeeper@865] -
>> > Initiating client connection, connectString=localhost:2181
>> > sessionTimeout=30000
>> > watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@28ba21f3
>> > 2017-09-19 10:03:31,397 [myid:] - INFO [main:ClientCnxnSocket@236] -
>> > jute.maxbuffer value is 4194304 Bytes
>> > 2017-09-19 10:03:31,410 [myid:localhost:2181] - INFO
>> > [main-SendThread(localhost:2181):ClientCnxn$SendThread@1113] - Opening
>> > socket connection to server localhost/127.0.0.1:2181. Will not attempt
>> > to authenticate using SASL (unknown error)
>> > Welcome to ZooKeeper!
>> > JLine support is enabled
>> > 2017-09-19 10:03:31,493 [myid:localhost:2181] - INFO
>> > [main-SendThread(localhost:2181):ClientCnxn$SendThread@948] - Socket
>> > connection established, initiating session, client: /127.0.0.1:44882,
>> > server: localhost/127.0.0.1:2181
>> > 2017-09-19 10:03:31,506 [myid:localhost:2181] - INFO
>> > [main-SendThread(localhost:2181):ClientCnxn$SendThread@1381] - Session
>> > establishment complete on server localhost/127.0.0.1:2181, sessionid =
>> > 0xeb0000480b3c005c, negotiated timeout = 30000
>> >
>> > WATCHER::
>> >
>> > WatchedEvent state:SyncConnected type:None path:null
>> > [zk: localhost:2181(CONNECTED) 0] create -e /testnode
>> > Created /testnode
>> > [zk: localhost:2181(CONNECTED) 1] stat /testnode
>> > cZxid = 0x2000002c4
>> > ctime = Tue Sep 19 10:03:50 CDT 2017
>> > mZxid = 0x2000002c4
>> > mtime = Tue Sep 19 10:03:50 CDT 2017
>> > pZxid = 0x2000002c4
>> > cversion = 0
>> > dataVersion = 0
>> > aclVersion = 0
>> > *ephemeralOwner = 0x0*
>> > dataLength = 0
>> > numChildren = 0
>> > [zk: localhost:2181(CONNECTED) 2]
>> > ============================================================
>> ==============================================
>> >
>> > If I disconnect and reconnect, the znode is still present as well. If I
>> > restart Zookeeper, many times this will resolve itself somehow and
>> > creation of ephemeral nodes will work correctly. I have also tried this
>> > in a clustered configuration and get cases where one Zookeeper will
>> > create ephemeral nodes correctly and another Zookeeper will exhibit the
>> > above issue.
>> >
>> > Please let me know what other information would be of use or other
>> > things I could be looking for.
>> >
>> > Any help appreciated.
>> >
>> > Mark
>> >
>> >
>> >
>>
>
>
Re: Problem creating Ephemeral znodes
Posted by Mark Johnson <ma...@servicemesh.com>.
Hi Abraham,
I believe I'm using nio - from the log:
2017-09-19 15:25:51,761 [myid:010000001020] - INFO
[main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with
10s sessionless connection timeout, 1 selector thread(s), 8 worker threads,
and 64 kB direct buffers.
And here's my zoo.cfg:
tickTime=2000
initLimit=100
syncLimit=5
dataDir=/var/lib/zookeeper
clientPort=2181
autopurge.snapRetainCount=3
# Purge task interval in hours
autopurge.purgeInterval=1
minSessionTimeout=4000
maxSessionTimeout=60000
standaloneEnabled=false
reconfigEnabled=true
skipACL=yes
dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic
*However, the problem is not removal of the ephemeral nodes, it's that
they're not getting created in the first place,* When I use the CLI to
create an ephemeral node via:
create -e /testnode
It's not marked as an ephemeral node and hangs around forever. From stat
/testnode:
ephemeralOwner = 0x0
Sometimes Zookeeper will come up and create ephemeral nodes and they work
correctly (deleted after the connection is terminated). But usually (with
the version I'm using- 3.5.3-beta2) they are not created.
Are you familiar with the code? I've downloaded the source and have been
looking for the place where the znodes are actually created but have not
found that section. If you can point me to the right area of the code I
can probably troubleshoot this issue.
Thanks for any light you can shed on this problem!
Mark
On Tue, Sep 19, 2017 at 6:18 PM, Abraham Fine <af...@apache.org> wrote:
> Hi Mark-
>
> I'm assuming you are using nio and not netty, please let me know if that
> is not the case.
>
> Have you possibly set zookeeper.nio.sessionlessCnxnTimeout? This value
> is 10s by default, so it may take a little more than ten seconds by
> default for ephemeral nodes to be deleted.
>
> Thanks,
> Abe
>
>
> On Tue, Sep 19, 2017, at 08:16, Mark Johnson wrote:
> > Hi,
> >
> > I'm running version 3.5.3-BETA of Zookeeper in a Docker container and
> > usually cannot create Ephemeral znodes, either through the CLI or though
> > a Java program. It appears the owner is not set. Here's an example
> > session:
> >
> > ============================================================
> ================================================
> > [root@Appliance ~]# docker exec -it 9b7cd1e80b5b /bin/bash
> > zookeeper@9b7cd1e80b5b:/opt/zookeeper-3.5.3-beta$ bin/zkCli.sh
> > Connecting to localhost:2181
> > 2017-09-19 10:03:31,373 [myid:] - INFO [main:Environment@109] - Client
> > environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b4
> 2864475a60,
> > built on 04/03/2017 16:19 GMT
> > 2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client
> > environment:host.name=9b7cd1e80b5b
> > 2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client
> > environment:java.version=1.8.0_144
> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> > environment:java.vendor=Oracle Corporation
> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> > environment:java.home=/usr/lib/jvm/java-8-oracle/jre
> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> > environment:java.class.path=/opt/zookeeper-3.5.3-beta/bin/.
> ./build/classes:/opt/zookeeper-3.5.3-beta/bin/../build/lib/*.jar:/opt/
> zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar
> > :/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.
> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Final.
> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/log4j-1.2.17.jar:/
> opt/zookeeper-3.5.3-beta/bin/../lib/jline-2.11.jar:/opt/
> zookeeper-3.5.3-beta/bin/..
> > /lib/jetty-util-9.2.18.v20160721.jar:/opt/zookeeper-
> 3.5.3-beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/
> opt/zookeeper-3.5.3-beta/bin/../lib/jetty-server-9.2.18.
> v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-
> security-9.2.18.v20160721.jar:/op
> > t/zookeeper-3.5.3-beta/bin/../lib/jetty-io-9.2.18.v20160721.
> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-http-9.2.18.
> v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/javax.
> servlet-api-3.1.0.jar:/opt/zookeeper-3.5.3-beta/bin/../
> lib/jackson-mapper-asl-1.
> > 9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jackson-core-
> asl-1.9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/commons-
> cli-1.2.jar:/opt/zookeeper-3.5.3-beta/bin/../zookeeper-3.5.
> 3-beta.jar:/opt/zookeeper-3.5.3-beta/bin/../src/java/lib/*.
> jar:/opt/zookeeper-
> > 3.5.3-beta/bin/../conf:
> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> > environment:java.library.path=/usr/java/packages/lib/amd64:/
> usr/lib64:/lib64:/lib:/usr/lib
> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> > environment:java.io.tmpdir=/tmp
> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> > environment:java.compiler=<NA>
> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> > environment:os.name=Linux
> > 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> > environment:os.arch=amd64
> > 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
> > environment:os.version=3.10.0-514.26.2.el7.x86_64
> > 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
> > environment:user.name=zookeeper
> > 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
> > environment:user.home=/home/zookeeper
> > 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
> > environment:user.dir=/opt/zookeeper-3.5.3-beta
> > 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
> > environment:os.memory.free=233MB
> > 2017-09-19 10:03:31,382 [myid:] - INFO [main:Environment@109] - Client
> > environment:os.memory.max=240MB
> > 2017-09-19 10:03:31,383 [myid:] - INFO [main:Environment@109] - Client
> > environment:os.memory.total=240MB
> > 2017-09-19 10:03:31,386 [myid:] - INFO [main:ZooKeeper@865] -
> > Initiating client connection, connectString=localhost:2181
> > sessionTimeout=30000
> > watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@28ba21f3
> > 2017-09-19 10:03:31,397 [myid:] - INFO [main:ClientCnxnSocket@236] -
> > jute.maxbuffer value is 4194304 Bytes
> > 2017-09-19 10:03:31,410 [myid:localhost:2181] - INFO
> > [main-SendThread(localhost:2181):ClientCnxn$SendThread@1113] - Opening
> > socket connection to server localhost/127.0.0.1:2181. Will not attempt
> > to authenticate using SASL (unknown error)
> > Welcome to ZooKeeper!
> > JLine support is enabled
> > 2017-09-19 10:03:31,493 [myid:localhost:2181] - INFO
> > [main-SendThread(localhost:2181):ClientCnxn$SendThread@948] - Socket
> > connection established, initiating session, client: /127.0.0.1:44882,
> > server: localhost/127.0.0.1:2181
> > 2017-09-19 10:03:31,506 [myid:localhost:2181] - INFO
> > [main-SendThread(localhost:2181):ClientCnxn$SendThread@1381] - Session
> > establishment complete on server localhost/127.0.0.1:2181, sessionid =
> > 0xeb0000480b3c005c, negotiated timeout = 30000
> >
> > WATCHER::
> >
> > WatchedEvent state:SyncConnected type:None path:null
> > [zk: localhost:2181(CONNECTED) 0] create -e /testnode
> > Created /testnode
> > [zk: localhost:2181(CONNECTED) 1] stat /testnode
> > cZxid = 0x2000002c4
> > ctime = Tue Sep 19 10:03:50 CDT 2017
> > mZxid = 0x2000002c4
> > mtime = Tue Sep 19 10:03:50 CDT 2017
> > pZxid = 0x2000002c4
> > cversion = 0
> > dataVersion = 0
> > aclVersion = 0
> > *ephemeralOwner = 0x0*
> > dataLength = 0
> > numChildren = 0
> > [zk: localhost:2181(CONNECTED) 2]
> > ============================================================
> ==============================================
> >
> > If I disconnect and reconnect, the znode is still present as well. If I
> > restart Zookeeper, many times this will resolve itself somehow and
> > creation of ephemeral nodes will work correctly. I have also tried this
> > in a clustered configuration and get cases where one Zookeeper will
> > create ephemeral nodes correctly and another Zookeeper will exhibit the
> > above issue.
> >
> > Please let me know what other information would be of use or other
> > things I could be looking for.
> >
> > Any help appreciated.
> >
> > Mark
> >
> >
> >
>
Re: Problem creating Ephemeral znodes
Posted by Abraham Fine <af...@apache.org>.
Hi Mark-
I'm assuming you are using nio and not netty, please let me know if that
is not the case.
Have you possibly set zookeeper.nio.sessionlessCnxnTimeout? This value
is 10s by default, so it may take a little more than ten seconds by
default for ephemeral nodes to be deleted.
Thanks,
Abe
On Tue, Sep 19, 2017, at 08:16, Mark Johnson wrote:
> Hi,
>
> I'm running version 3.5.3-BETA of Zookeeper in a Docker container and
> usually cannot create Ephemeral znodes, either through the CLI or though
> a Java program. It appears the owner is not set. Here's an example
> session:
>
> ============================================================================================================
> [root@Appliance ~]# docker exec -it 9b7cd1e80b5b /bin/bash
> zookeeper@9b7cd1e80b5b:/opt/zookeeper-3.5.3-beta$ bin/zkCli.sh
> Connecting to localhost:2181
> 2017-09-19 10:03:31,373 [myid:] - INFO [main:Environment@109] - Client
> environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60,
> built on 04/03/2017 16:19 GMT
> 2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client
> environment:host.name=9b7cd1e80b5b
> 2017-09-19 10:03:31,378 [myid:] - INFO [main:Environment@109] - Client
> environment:java.version=1.8.0_144
> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> environment:java.vendor=Oracle Corporation
> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> environment:java.class.path=/opt/zookeeper-3.5.3-beta/bin/../build/classes:/opt/zookeeper-3.5.3-beta/bin/../build/lib/*.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar
> :/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/log4j-1.2.17.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jline-2.11.jar:/opt/zookeeper-3.5.3-beta/bin/..
> /lib/jetty-util-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-server-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-security-9.2.18.v20160721.jar:/op
> t/zookeeper-3.5.3-beta/bin/../lib/jetty-io-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-http-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jackson-mapper-asl-1.
> 9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jackson-core-asl-1.9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/commons-cli-1.2.jar:/opt/zookeeper-3.5.3-beta/bin/../zookeeper-3.5.3-beta.jar:/opt/zookeeper-3.5.3-beta/bin/../src/java/lib/*.jar:/opt/zookeeper-
> 3.5.3-beta/bin/../conf:
> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> environment:java.io.tmpdir=/tmp
> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> environment:java.compiler=<NA>
> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> environment:os.name=Linux
> 2017-09-19 10:03:31,380 [myid:] - INFO [main:Environment@109] - Client
> environment:os.arch=amd64
> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
> environment:os.version=3.10.0-514.26.2.el7.x86_64
> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
> environment:user.name=zookeeper
> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
> environment:user.home=/home/zookeeper
> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
> environment:user.dir=/opt/zookeeper-3.5.3-beta
> 2017-09-19 10:03:31,381 [myid:] - INFO [main:Environment@109] - Client
> environment:os.memory.free=233MB
> 2017-09-19 10:03:31,382 [myid:] - INFO [main:Environment@109] - Client
> environment:os.memory.max=240MB
> 2017-09-19 10:03:31,383 [myid:] - INFO [main:Environment@109] - Client
> environment:os.memory.total=240MB
> 2017-09-19 10:03:31,386 [myid:] - INFO [main:ZooKeeper@865] -
> Initiating client connection, connectString=localhost:2181
> sessionTimeout=30000
> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@28ba21f3
> 2017-09-19 10:03:31,397 [myid:] - INFO [main:ClientCnxnSocket@236] -
> jute.maxbuffer value is 4194304 Bytes
> 2017-09-19 10:03:31,410 [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1113] - Opening
> socket connection to server localhost/127.0.0.1:2181. Will not attempt
> to authenticate using SASL (unknown error)
> Welcome to ZooKeeper!
> JLine support is enabled
> 2017-09-19 10:03:31,493 [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@948] - Socket
> connection established, initiating session, client: /127.0.0.1:44882,
> server: localhost/127.0.0.1:2181
> 2017-09-19 10:03:31,506 [myid:localhost:2181] - INFO
> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1381] - Session
> establishment complete on server localhost/127.0.0.1:2181, sessionid =
> 0xeb0000480b3c005c, negotiated timeout = 30000
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
> [zk: localhost:2181(CONNECTED) 0] create -e /testnode
> Created /testnode
> [zk: localhost:2181(CONNECTED) 1] stat /testnode
> cZxid = 0x2000002c4
> ctime = Tue Sep 19 10:03:50 CDT 2017
> mZxid = 0x2000002c4
> mtime = Tue Sep 19 10:03:50 CDT 2017
> pZxid = 0x2000002c4
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> *ephemeralOwner = 0x0*
> dataLength = 0
> numChildren = 0
> [zk: localhost:2181(CONNECTED) 2]
> ==========================================================================================================
>
> If I disconnect and reconnect, the znode is still present as well. If I
> restart Zookeeper, many times this will resolve itself somehow and
> creation of ephemeral nodes will work correctly. I have also tried this
> in a clustered configuration and get cases where one Zookeeper will
> create ephemeral nodes correctly and another Zookeeper will exhibit the
> above issue.
>
> Please let me know what other information would be of use or other
> things I could be looking for.
>
> Any help appreciated.
>
> Mark
>
>
>