You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@zookeeper.apache.org by "Rounak Kakkar (Jira)" <ji...@apache.org> on 2025/02/10 06:29:00 UTC
[jira] [Created] (ZOOKEEPER-4896) Inconsistency for a ZK Path in the cluster
Rounak Kakkar created ZOOKEEPER-4896:
----------------------------------------
Summary: Inconsistency for a ZK Path in the cluster
Key: ZOOKEEPER-4896
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4896
Project: ZooKeeper
Issue Type: Bug
Affects Versions: 3.8.4
Environment: Zookeeper is deployed in the cluster environment. There are three nodes in the cluster. Attached the config
Reporter: Rounak Kakkar
Attachments: zk_config_stats, zk_path_issue
We are using Zookeeper in cluster mode. For one of the paths, we notice a data inconsistency between a follower node and the other nodes in the system. We observe that the version for the path increases alongside content updates. However, there are four additional entries in that path in the follower where the issue is present. The version consistently lags four updates behind both the leader and the other followers.
If we try to delete one of the extra entries from that follower node, it fails with the error: Node does not exist.
*Leader:*
{{{}vmanage@60f9084accc9:/var/lib/zookeeper$ /var/lib/zookeeper/bin/zkServer.sh status{}}}{{{}ZooKeeper JMX enabled by default{}}}{{{}Using config: /var/lib/zookeeper/bin/../conf/zoo.cfg{}}}{{{}Client port found: 2181. Client address: localhost. Client SSL: false.{}}}{{{}Mode: leader{}}}{{{}vmanage@60f9084accc9:/var/lib/zookeeper${}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 1] stat -w /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}cZxid = 0x13000611b0{}}}{{{}ctime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}mZxid = 0x13000611b0{}}}{{{}mtime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}pZxid = 0x9109c98fb2{}}}{{{}cversion = 2332172{}}}{{{}dataVersion = 0{}}}{{{}aclVersion = 0{}}}{{{}ephemeralOwner = 0x0{}}}{{{}dataLength = 0{}}}{{{}numChildren = 22{}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 2] ls /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}[20250206_0_685_142, 20250206_0_690_143, 20250206_0_695_144, 20250206_0_700_145, 20250206_686_686_0, 20250206_687_687_0, 20250206_688_688_0, 20250206_689_689_0, 20250206_690_690_0, 20250206_691_691_0, 20250206_692_692_0, 20250206_693_693_0, 20250206_694_694_0, 20250206_695_695_0, 20250206_696_696_0, 20250206_697_697_0, 20250206_698_698_0, 20250206_699_699_0, 20250206_700_700_0, 20250206_701_701_0, 20250206_702_702_0, 20250206_703_703_0]{}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 3] stat -w /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}cZxid = 0x13000611b0{}}}{{{}ctime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}mZxid = 0x13000611b0{}}}{{{}mtime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}pZxid = 0x9109c9b371{}}}{{{}cversion = 2332178{}}}{{{}dataVersion = 0{}}}{{{}aclVersion = 0{}}}{{{}ephemeralOwner = 0x0{}}}{{{}dataLength = 0{}}}{{{}numChildren = 26{}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 4] ls /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}[20250206_0_685_142, 20250206_0_690_143, 20250206_0_695_144, 20250206_0_700_145, 20250206_0_705_146, 20250206_686_686_0, 20250206_687_687_0, 20250206_688_688_0, 20250206_689_689_0, 20250206_690_690_0, 20250206_691_691_0, 20250206_692_692_0, 20250206_693_693_0, 20250206_694_694_0, 20250206_695_695_0, 20250206_696_696_0, 20250206_697_697_0, 20250206_698_698_0, 20250206_699_699_0, 20250206_700_700_0, 20250206_701_701_0, 20250206_702_702_0, 20250206_703_703_0, 20250206_704_704_0, 20250206_705_705_0, 20250206_706_706_0]{}}}
*Follower-1 (No issue seen):*
{{{}vmanage@aacf5ef6555b:/var/lib/zookeeper$ /var/lib/zookeeper/bin/zkServer.sh status{}}}{{{}ZooKeeper JMX enabled by default{}}}{{{}Using config: /var/lib/zookeeper/bin/../conf/zoo.cfg{}}}{{{}Client port found: 2181. Client address: localhost. Client SSL: false.{}}}{{{}Mode: follower{}}}{{{}vmanage@aacf5ef6555b:/var/lib/zookeeper${}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 1] stat -w /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}cZxid = 0x13000611b0{}}}{{{}ctime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}mZxid = 0x13000611b0{}}}{{{}mtime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}pZxid = 0x9109c98fb2{}}}{{{}cversion = 2332172{}}}{{{}dataVersion = 0{}}}{{{}aclVersion = 0{}}}{{{}ephemeralOwner = 0x0{}}}{{{}dataLength = 0{}}}{{{}numChildren = 22{}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 2] ls /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}[20250206_0_685_142, 20250206_0_690_143, 20250206_0_695_144, 20250206_0_700_145, 20250206_686_686_0, 20250206_687_687_0, 20250206_688_688_0, 20250206_689_689_0, 20250206_690_690_0, 20250206_691_691_0, 20250206_692_692_0, 20250206_693_693_0, 20250206_694_694_0, 20250206_695_695_0, 20250206_696_696_0, 20250206_697_697_0, 20250206_698_698_0, 20250206_699_699_0, 20250206_700_700_0, 20250206_701_701_0, 20250206_702_702_0, 20250206_703_703_0]{}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 3] stat -w /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}cZxid = 0x13000611b0{}}}{{{}ctime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}mZxid = 0x13000611b0{}}}{{{}mtime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}pZxid = 0x9109c9b371{}}}{{{}cversion = 2332178{}}}{{{}dataVersion = 0{}}}{{{}aclVersion = 0{}}}{{{}ephemeralOwner = 0x0{}}}{{{}dataLength = 0{}}}{{{}numChildren = 26{}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 4] ls /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}[20250206_0_685_142, 20250206_0_690_143, 20250206_0_695_144, 20250206_0_700_145, 20250206_0_705_146, 20250206_686_686_0, 20250206_687_687_0, 20250206_688_688_0, 20250206_689_689_0, 20250206_690_690_0, 20250206_691_691_0, 20250206_692_692_0, 20250206_693_693_0, 20250206_694_694_0, 20250206_695_695_0, 20250206_696_696_0, 20250206_697_697_0, 20250206_698_698_0, 20250206_699_699_0, 20250206_700_700_0, 20250206_701_701_0, 20250206_702_702_0, 20250206_703_703_0, 20250206_704_704_0, 20250206_705_705_0, 20250206_706_706_0]{}}}
*Follower-2 (Issue seen):*
{{{}vmanage@fc890e903696:/var/lib/zookeeper$ /var/lib/zookeeper/bin/zkServer.sh status{}}}{{{}ZooKeeper JMX enabled by default{}}}{{{}Using config: /var/lib/zookeeper/bin/../conf/zoo.cfg{}}}{{{}Client port found: 2181. Client address: localhost. Client SSL: false.{}}}{{{}Mode: follower{}}}{{{}vmanage@fc890e903696:/var/lib/zookeeper${}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 1] stat -w /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}cZxid = 0x13000611b0{}}}{{{}ctime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}mZxid = 0x13000611b0{}}}{{{}mtime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}pZxid = 0x9109c98fb2{}}}{{{}cversion = 2332168{}}}{{{}dataVersion = 0{}}}{{{}aclVersion = 0{}}}{{{}ephemeralOwner = 0x0{}}}{{{}dataLength = 0{}}}{{{}numChildren = 26{}}}{{{}[zk: 127.0.0.1:2181(CONNECTED) 2]{}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 2] ls /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}[20250130_0_4587_333, 20250130_0_4592_334, 20250130_4443_4565_11, 20250130_4560_4560_0, 20250206_0_685_142, 20250206_0_690_143, 20250206_0_695_144, 20250206_0_700_145, 20250206_686_686_0, 20250206_687_687_0, 20250206_688_688_0, 20250206_689_689_0, 20250206_690_690_0, 20250206_691_691_0, 20250206_692_692_0, 20250206_693_693_0, 20250206_694_694_0, 20250206_695_695_0, 20250206_696_696_0, 20250206_697_697_0, 20250206_698_698_0, 20250206_699_699_0, 20250206_700_700_0, 20250206_701_701_0, 20250206_702_702_0, 20250206_703_703_0]{}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 3] stat -w /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}cZxid = 0x13000611b0{}}}{{{}ctime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}mZxid = 0x13000611b0{}}}{{{}mtime = Thu Jun 06 13:04:52 UTC 2024{}}}{{{}pZxid = 0x9109c9b371{}}}{{{}cversion = 2332174{}}}{{{}dataVersion = 0{}}}{{{}aclVersion = 0{}}}{{{}ephemeralOwner = 0x0{}}}{{{}dataLength = 0{}}}{{{}numChildren = 30{}}}{{{}[zk: 127.0.0.1:2181(CONNECTED) 4]{}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 4] ls /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts{}}}{{{}[20250130_0_4587_333, 20250130_0_4592_334, 20250130_4443_4565_11, 20250130_4560_4560_0, 20250206_0_685_142, 20250206_0_690_143, 20250206_0_695_144, 20250206_0_700_145, 20250206_0_705_146, 20250206_686_686_0, 20250206_687_687_0, 20250206_688_688_0, 20250206_689_689_0, 20250206_690_690_0, 20250206_691_691_0, 20250206_692_692_0, 20250206_693_693_0, 20250206_694_694_0, 20250206_695_695_0, 20250206_696_696_0, 20250206_697_697_0, 20250206_698_698_0, 20250206_699_699_0, 20250206_700_700_0, 20250206_701_701_0, 20250206_702_702_0, 20250206_703_703_0, 20250206_704_704_0, 20250206_705_705_0, 20250206_706_706_0]{}}}
{{{}[zk: 127.0.0.1:2181(CONNECTED) 5] delete /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts/20250130_0_4587_333{}}}{{{}Node does not exist: /clickhouse/tables/shard_2/aggregated_apps_dpi_app_summary_7f1396c5_a4d8_48c9_a3f3_90271305c114/replicas/replica_2/parts/20250130_0_4587_333{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)