You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@curator.apache.org by "Francis Simon (Jira)" <ji...@apache.org> on 2022/05/03 12:03:00 UTC
[jira] [Comment Edited] (CURATOR-638) Curator disconnect from zookeeper when IPs change
[ https://issues.apache.org/jira/browse/CURATOR-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17531159#comment-17531159 ]
Francis Simon edited comment on CURATOR-638 at 5/3/22 12:02 PM:
----------------------------------------------------------------
I was able to reproduce the issue with a bare bones example which is attached to the ticket. I took the curator service discovery example and basically just wrapped it in docker. Again we haven no trouble with zookeeper itself, it is just the curator clients. In Kubernetes we use the [Bitnami Zookeeper Chart]([https://github.com/bitnami/charts/tree/master/bitnami/zookeeper)] which is a statefulset.
was (Author: JIRAUSER288911):
I was able to reproduce the issue with a bare bones example which is attached to the ticket. I took the curator service discovery example and basically just wrapped it in docker. Again we haven bo trouble with zookeeper itself, it is just the curator clients. In Kubernetes we use the [Bitnami Zookeeper Chart]([https://github.com/bitnami/charts/tree/master/bitnami/zookeeper)] which is a statefulset.
> Curator disconnect from zookeeper when IPs change
> -------------------------------------------------
>
> Key: CURATOR-638
> URL: https://issues.apache.org/jira/browse/CURATOR-638
> Project: Apache Curator
> Issue Type: Bug
> Components: Client, Recipes
> Affects Versions: 5.2.1
> Environment: Docker or Kubernetes, docker example provided
> Reporter: Francis Simon
> Priority: Blocker
> Attachments: zkissue.zip
>
>
> Blocking usage of Zookeeper in production. Tried testing a few versions all had the issue. Effects any recipes that use ephemeral nodes. Example attached.
> We use multiple Apache Curator recipes in our system which is running in Docker and Kubernetes. The behavior I am seeing is that curator appears to resolve to the IP address of the containers rather than being tied to DNS names. I have seen old tickets on this, but the behavior is reproducible on the latest code release.
> We are running zookeeper in containers on kubernetes. In kubernetes many things could cause a container to move hosts, the pod disruption budget ensures that a quorum is always present. But with this bug if all nodes move for any reason and get new IP addresses clients will disconnect when they shouldn't. Disconnecting has the bad side effect that all ephemeral nodes are lost. This effects for us coordination, distributed locking and service discovery. Causes production downtime so marked as a Blocker.
> I have a simple sample which just uses the service discovery recipe to register a bunch of services in zookeeper. I run the example in docker compose. It is 100% reproducible.
>
> {code:java}
> # Standup zookeeper and wait for it to be healthy
> docker-compose up -d zookeeper1 zookeeper2 zookeeper3
> # Stand up a server and make sure it is connected and working as expected
> docker-compose up -d server1
> # Take down a single zookeeper node and stand up another agent.
> # The agent will grab the old zookeepers IP address
> docker-compose rm -s zookeeper1
> docker-compose up -d server2
> # Bring the zookeeper node back up.
> # Wait for it be healthy
> docker-compose up -d zookeeper1
> # Then take down the next zookeeper node and stand up another agent.
> # The agent will grab the old zookeepers IP address
> docker-compose rm -s zookeeper2
> docker-compose up -d server3
> # Bring the zookeeper node back up.
> # Wait for it be healthy
> docker-compose up -d zookeeper2
> # Then take down the next zookeeper node and stand up another agent.
> # The agent will grab the old zookeepers IP address
> docker-compose rm -s zookeeper3
> docker-compose up -d server4
> # Bring the zookeeper node back up.
> # Wait for it be healthy
> docker-compose up -d zookeeper3{code}
>
> At the time of taking down the 3rd zookeeper node, the first server1 that was stood up will now receive a disconnected status because the IP of all three nodes has no changes form the original IP addresses.
>
> {code:java}
> server1_1 | Query instances for servicetest
> server1_1 | Exception in thread "main" java.lang.RuntimeException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /myservices/test/62e23a0b-dfdb-46f5-966f-8dc7a4978c70
> server1_1 | at org.apache.curator.shaded.com.google.common.base.Throwables.propagate(Throwables.java:241)
> server1_1 | at org.apache.curator.utils.ExceptionAccumulator.propagate(ExceptionAccumulator.java:38)
> server1_1 | at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.close(ServiceDiscoveryImpl.java:171)
> server1_1 | at org.apache.curator.shaded.com.google.common.io.Closeables.close(Closeables.java:78)
> server1_1 | at org.apache.curator.utils.CloseableUtils.closeQuietly(CloseableUtils.java:59)
> server1_1 | at zkissue.App.main(App.java:72)
> server1_1 | Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /myservices/test/62e23a0b-dfdb-46f5-966f-8dc7a4978c70
> server1_1 | at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
> server1_1 | at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
> server1_1 | at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:2001)
> server1_1 | at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:274)
> server1_1 | at org.apache.curator.framework.imps.DeleteBuilderImpl$5.call(DeleteBuilderImpl.java:268)
> server1_1 | at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:93)
> server1_1 | at org.apache.curator.framework.imps.DeleteBuilderImpl.pathInForeground(DeleteBuilderImpl.java:265)
> server1_1 | at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:249)
> server1_1 | at org.apache.curator.framework.imps.DeleteBuilderImpl.forPath(DeleteBuilderImpl.java:34)
> server1_1 | at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.internalUnregisterService(ServiceDiscoveryImpl.java:520)
> server1_1 | at org.apache.curator.x.discovery.details.ServiceDiscoveryImpl.close(ServiceDiscoveryImpl.java:157)
> server1_1 | ... 3 more
> {code}
>
> This causes it to disconnect and lose its discovery state which can be seen from the other services.
> {code:java}
> server2_1 | Query instances for servicetest
> server2_1 | test
> server2_1 | service description: http://server-4:57456
> server2_1 | service description: http://server-3:37740
> server2_1 | service description: http://server-2:40219{code}
>
> Should mention that the Zookeeper cluster is always happy and healthy. This is a client side issue.
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)