You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by arjun s v <ar...@gmail.com> on 2025/06/19 13:05:35 UTC
Ephemeral Node Session ID Mismatch in ZooKeeper for a Rejoined Server
Team,
I'm investigating an issue where an ephemeral node in ZooKeeper was not
properly managed after a server rejoined the ensemble. My setup uses
ZooKeeper 3.9.3. Below is the timeline of events:
Timeline:
1. An ephemeral node is created.
2. This is synced across all servers in the ensemble.
3. Follower 'A' goes out of the ensemble due to a connectivity issue.
4. Now the client session associated with the ephemeral node
disconnects, deleting the ephemeral node across all active servers in the
ensemble.
5. A new client session is initiated, creating another ephemeral node
with the same path.
6. This new ephemeral node is synced across all active servers in the
ensemble.
7. Follower 'A' rejoins the ensemble.
8. The leader syncs the latest commits to follower 'A'.
9. However, (Ephemeral Node).getEphemeralOwner() does not return the
current session's session ID.
I couldn't confirm if an old ephemeral node persisted, as the machine was
restarted, resolving the issue. Debug logs were not enabled, so no
additional logs are available to confirm the root cause. I suspect packet
loss during the rejoin may have contributed. Attached are the
leader-to-follower sync logs.
Could you please advise if there are known issues with ephemeral node
cleanup during server rejoins, or other scenarios to check? Is this likely
due to packet loss or synchronization issues?
Thanks in advance!
Sync logs:
> Follower
>
> 11:29:25:853 org.apache.zookeeper.server.quorum.Learner$LeaderConnector.run
> Successfully connected to leader, using address: <DOMAIN/IPADDR>:2888
>
> 11:29:25:854 org.apache.zookeeper.util.SecurityUtils.createSaslClient
> QuorumLearner will use DIGEST-MD5 as SASL mechanism.
>
> 11:29:25:859
> org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.checkAuthStatus
> Successfully completed the authentication using SASL. server addr:
> <DOMAIN/IPADDR>:2888, status: SUCCESS
>
> 11:29:25:864 org.apache.zookeeper.server.quorum.QuorumPeer.setZabState
> Peer state changed: following - synchronization
>
> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> Getting a diff from the leader 0x240025c7cc
>
> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> Got zxid 0x240025c7cc expected 0x1
>
> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> Learner received NEWLEADER message
>
> 11:29:25:869 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
> Peer state changed: following - synchronization - diff
>
> 11:29:25:870
> org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier
> Dynamic reconfig is disabled, we don't store the last seen config.
>
> 11:29:25:871 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> It took 2ms to persist and commit txns in packetsCommitted. 0 outstanding
> txns left in packetsNotLogged
>
> 11:29:25:873 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> Set the current epoch to 37
>
> 11:29:25:874 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
> Peer state changed: following - synchronization
>
> 11:29:25:874 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> Sent NEWLEADER ack to leader with zxid 2500000000
>
> 11:29:25:879 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> Learner received UPTODATE message
>
>
>
> Leader:
>
> 11:29:25:864 org.apache.zookeeper.server.quorum.LearnerHandler.run
> Follower sid: 2 : info : <ADDR>:2888:3888:participant
>
> 11:29:25:868 org.apache.zookeeper.server.ZKDatabase.isTxnLogSyncEnabled
> On disk txn sync enabled with snapshotSizeFactor 0.33
>
> 11:29:25:868 org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
> Synchronizing with Learner sid: 2 maxCommittedLog=0x240025c7cc
> minCommittedLog=0x240025c5d0 lastProcessedZxid=0x240025c7cc
> peerLastZxid=0x240025c7c3
>
> 11:29:25:869 org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
> Using committedLog for peer sid: 2
>
> 11:29:25:870
> org.apache.zookeeper.server.quorum.LearnerHandler.queueCommittedProposals
> Sending DIFF zxid=0x240025c7cc for peer sid: 2
Re: Ephemeral Node Session ID Mismatch in ZooKeeper for a Rejoined Server
Posted by arjun s v <ar...@gmail.com>.
Also could see 3.8.4 is affected by CVE-2024-47554.
On Tue, Jun 24, 2025 at 10:33 AM arjun s v <ar...@gmail.com> wrote:
> Andor,
>
> Thank you for your response.
>
> We are uncertain about the specific conditions triggering the issue,
> making it difficult to predict its occurrence in version 3.8.4.
>
> As multiple critical modules, including Hadoop and Kafka, rely on
> Zookeeper, any disruption could lead to potential data loss on our end.
>
> Identifying the exact root cause or scenario would help us both to either
> fix or mitigate the issue.
>
> I will provide any additional logs if needed.
>
> Could you please clarify the specific issue and confirm if a fix is
> available in version 3.8.4?
>
> On Mon, Jun 23, 2025 at 9:27 PM Andor Molnar <an...@apache.org> wrote:
>
>> Hi Arjun,
>>
>> Could you please validate the same scenario with latest stable version
>> 3.8.4?
>>
>> Andor
>>
>>
>>
>>
>> > On Jun 23, 2025, at 00:09, arjun s v <ar...@gmail.com> wrote:
>> >
>> > Continuation to the Ephemeral node issue,
>> >
>> > I observed that the learner sends ACKs for each packet it receives, but
>> > there seems to be no verification on the leader's side to confirm these
>> > ACKs against the packets sent.
>> > Is there a configuration option that, when enabled, ensures all packet
>> > ACKs, including COMMIT ACKs, are validated?
>> > If packet loss is the reason for this issue, verifying all received ACKs
>> > against the sent packets could help prevent such problems in the future.
>> >
>> > Please advise.
>> >
>> > On Thu, Jun 19, 2025 at 6:35 PM arjun s v <ar...@gmail.com>
>> wrote:
>> >
>> >> Team,
>> >>
>> >> I'm investigating an issue where an ephemeral node in ZooKeeper was not
>> >> properly managed after a server rejoined the ensemble. My setup uses
>> >> ZooKeeper 3.9.3. Below is the timeline of events:
>> >>
>> >> Timeline:
>> >>
>> >> 1. An ephemeral node is created.
>> >> 2. This is synced across all servers in the ensemble.
>> >> 3. Follower 'A' goes out of the ensemble due to a connectivity issue.
>> >> 4. Now the client session associated with the ephemeral node
>> >> disconnects, deleting the ephemeral node across all active servers in
>> the
>> >> ensemble.
>> >> 5. A new client session is initiated, creating another ephemeral node
>> >> with the same path.
>> >> 6. This new ephemeral node is synced across all active servers in the
>> >> ensemble.
>> >> 7. Follower 'A' rejoins the ensemble.
>> >> 8. The leader syncs the latest commits to follower 'A'.
>> >> 9. However, (Ephemeral Node).getEphemeralOwner() does not return the
>> >> current session's session ID.
>> >>
>> >>
>> >> I couldn't confirm if an old ephemeral node persisted, as the machine
>> was
>> >> restarted, resolving the issue. Debug logs were not enabled, so no
>> >> additional logs are available to confirm the root cause. I suspect
>> packet
>> >> loss during the rejoin may have contributed. Attached are the
>> >> leader-to-follower sync logs.
>> >>
>> >> Could you please advise if there are known issues with ephemeral node
>> >> cleanup during server rejoins, or other scenarios to check? Is this
>> likely
>> >> due to packet loss or synchronization issues?
>> >>
>> >> Thanks in advance!
>> >>
>> >>
>> >> Sync logs:
>> >>
>> >>> Follower
>> >>>
>> >>> 11:29:25:853
>> >>> org.apache.zookeeper.server.quorum.Learner$LeaderConnector.run
>> >>> Successfully connected to leader, using address: <DOMAIN/IPADDR>:2888
>> >>>
>> >>> 11:29:25:854 org.apache.zookeeper.util.SecurityUtils.createSaslClient
>> >>> QuorumLearner will use DIGEST-MD5 as SASL mechanism.
>> >>>
>> >>> 11:29:25:859
>> >>>
>> org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.checkAuthStatus
>> >>> Successfully completed the authentication using SASL. server addr:
>> >>> <DOMAIN/IPADDR>:2888, status: SUCCESS
>> >>>
>> >>> 11:29:25:864 org.apache.zookeeper.server.quorum.QuorumPeer.setZabState
>> >>> Peer state changed: following - synchronization
>> >>>
>> >>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> >>> Getting a diff from the leader 0x240025c7cc
>> >>>
>> >>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> >>> Got zxid 0x240025c7cc expected 0x1
>> >>>
>> >>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> >>> Learner received NEWLEADER message
>> >>>
>> >>> 11:29:25:869 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
>> >>> Peer state changed: following - synchronization - diff
>> >>>
>> >>> 11:29:25:870
>> >>>
>> org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier
>> >>> Dynamic reconfig is disabled, we don't store the last seen config.
>> >>>
>> >>> 11:29:25:871 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> >>> It took 2ms to persist and commit txns in packetsCommitted. 0
>> outstanding
>> >>> txns left in packetsNotLogged
>> >>>
>> >>> 11:29:25:873 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> >>> Set the current epoch to 37
>> >>>
>> >>> 11:29:25:874 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
>> >>> Peer state changed: following - synchronization
>> >>>
>> >>> 11:29:25:874 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> >>> Sent NEWLEADER ack to leader with zxid 2500000000
>> >>>
>> >>> 11:29:25:879 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> >>> Learner received UPTODATE message
>> >>>
>> >>>
>> >>>
>> >>> Leader:
>> >>>
>> >>> 11:29:25:864 org.apache.zookeeper.server.quorum.LearnerHandler.run
>> >>> Follower sid: 2 : info : <ADDR>:2888:3888:participant
>> >>>
>> >>> 11:29:25:868
>> org.apache.zookeeper.server.ZKDatabase.isTxnLogSyncEnabled
>> >>> On disk txn sync enabled with snapshotSizeFactor 0.33
>> >>>
>> >>> 11:29:25:868
>> >>> org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
>> >>> Synchronizing with Learner sid: 2 maxCommittedLog=0x240025c7cc
>> >>> minCommittedLog=0x240025c5d0 lastProcessedZxid=0x240025c7cc
>> >>> peerLastZxid=0x240025c7c3
>> >>>
>> >>> 11:29:25:869
>> >>> org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
>> >>> Using committedLog for peer sid: 2
>> >>>
>> >>> 11:29:25:870
>> >>>
>> org.apache.zookeeper.server.quorum.LearnerHandler.queueCommittedProposals
>> >>> Sending DIFF zxid=0x240025c7cc for peer sid: 2
>> >>
>> >>
>>
>>
Re: Ephemeral Node Session ID Mismatch in ZooKeeper for a Rejoined Server
Posted by Andor Molnar <an...@apache.org>.
Hi Arjun,
Yes, I understand your concerns, just tried to suggest an approach which could realistically bring you closer to the resolution. There were a lot of changes in the 3.9.x release line around this area and verifying 3.8.4 would help us rule out where the problem could be.
What you can do is to upload server logs from all your ZooKeeper nodes, but I cannot tell you if anybody from here will be able to take a look at them in short term especially in the summer season.
ZooKeeper 3.8.4 is still the latest stable version of ZooKeeper and recommended for production usage. The CVE you mentioned is valid, we should create a new version as soon as possible.
Andor
> On Jun 24, 2025, at 00:03, arjun s v <ar...@gmail.com> wrote:
>
> Andor,
>
> Thank you for your response.
>
> We are uncertain about the specific conditions triggering the issue, making
> it difficult to predict its occurrence in version 3.8.4.
>
> As multiple critical modules, including Hadoop and Kafka, rely on
> Zookeeper, any disruption could lead to potential data loss on our end.
>
> Identifying the exact root cause or scenario would help us both to either
> fix or mitigate the issue.
>
> I will provide any additional logs if needed.
>
> Could you please clarify the specific issue and confirm if a fix is
> available in version 3.8.4?
>
> On Mon, Jun 23, 2025 at 9:27 PM Andor Molnar <an...@apache.org> wrote:
>
>> Hi Arjun,
>>
>> Could you please validate the same scenario with latest stable version
>> 3.8.4?
>>
>> Andor
>>
>>
>>
>>
>>> On Jun 23, 2025, at 00:09, arjun s v <ar...@gmail.com> wrote:
>>>
>>> Continuation to the Ephemeral node issue,
>>>
>>> I observed that the learner sends ACKs for each packet it receives, but
>>> there seems to be no verification on the leader's side to confirm these
>>> ACKs against the packets sent.
>>> Is there a configuration option that, when enabled, ensures all packet
>>> ACKs, including COMMIT ACKs, are validated?
>>> If packet loss is the reason for this issue, verifying all received ACKs
>>> against the sent packets could help prevent such problems in the future.
>>>
>>> Please advise.
>>>
>>> On Thu, Jun 19, 2025 at 6:35 PM arjun s v <ar...@gmail.com> wrote:
>>>
>>>> Team,
>>>>
>>>> I'm investigating an issue where an ephemeral node in ZooKeeper was not
>>>> properly managed after a server rejoined the ensemble. My setup uses
>>>> ZooKeeper 3.9.3. Below is the timeline of events:
>>>>
>>>> Timeline:
>>>>
>>>> 1. An ephemeral node is created.
>>>> 2. This is synced across all servers in the ensemble.
>>>> 3. Follower 'A' goes out of the ensemble due to a connectivity issue.
>>>> 4. Now the client session associated with the ephemeral node
>>>> disconnects, deleting the ephemeral node across all active servers in
>> the
>>>> ensemble.
>>>> 5. A new client session is initiated, creating another ephemeral node
>>>> with the same path.
>>>> 6. This new ephemeral node is synced across all active servers in the
>>>> ensemble.
>>>> 7. Follower 'A' rejoins the ensemble.
>>>> 8. The leader syncs the latest commits to follower 'A'.
>>>> 9. However, (Ephemeral Node).getEphemeralOwner() does not return the
>>>> current session's session ID.
>>>>
>>>>
>>>> I couldn't confirm if an old ephemeral node persisted, as the machine
>> was
>>>> restarted, resolving the issue. Debug logs were not enabled, so no
>>>> additional logs are available to confirm the root cause. I suspect
>> packet
>>>> loss during the rejoin may have contributed. Attached are the
>>>> leader-to-follower sync logs.
>>>>
>>>> Could you please advise if there are known issues with ephemeral node
>>>> cleanup during server rejoins, or other scenarios to check? Is this
>> likely
>>>> due to packet loss or synchronization issues?
>>>>
>>>> Thanks in advance!
>>>>
>>>>
>>>> Sync logs:
>>>>
>>>>> Follower
>>>>>
>>>>> 11:29:25:853
>>>>> org.apache.zookeeper.server.quorum.Learner$LeaderConnector.run
>>>>> Successfully connected to leader, using address: <DOMAIN/IPADDR>:2888
>>>>>
>>>>> 11:29:25:854 org.apache.zookeeper.util.SecurityUtils.createSaslClient
>>>>> QuorumLearner will use DIGEST-MD5 as SASL mechanism.
>>>>>
>>>>> 11:29:25:859
>>>>>
>> org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.checkAuthStatus
>>>>> Successfully completed the authentication using SASL. server addr:
>>>>> <DOMAIN/IPADDR>:2888, status: SUCCESS
>>>>>
>>>>> 11:29:25:864 org.apache.zookeeper.server.quorum.QuorumPeer.setZabState
>>>>> Peer state changed: following - synchronization
>>>>>
>>>>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>>>> Getting a diff from the leader 0x240025c7cc
>>>>>
>>>>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>>>> Got zxid 0x240025c7cc expected 0x1
>>>>>
>>>>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>>>> Learner received NEWLEADER message
>>>>>
>>>>> 11:29:25:869 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
>>>>> Peer state changed: following - synchronization - diff
>>>>>
>>>>> 11:29:25:870
>>>>> org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier
>>>>> Dynamic reconfig is disabled, we don't store the last seen config.
>>>>>
>>>>> 11:29:25:871 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>>>> It took 2ms to persist and commit txns in packetsCommitted. 0
>> outstanding
>>>>> txns left in packetsNotLogged
>>>>>
>>>>> 11:29:25:873 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>>>> Set the current epoch to 37
>>>>>
>>>>> 11:29:25:874 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
>>>>> Peer state changed: following - synchronization
>>>>>
>>>>> 11:29:25:874 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>>>> Sent NEWLEADER ack to leader with zxid 2500000000
>>>>>
>>>>> 11:29:25:879 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>>>> Learner received UPTODATE message
>>>>>
>>>>>
>>>>>
>>>>> Leader:
>>>>>
>>>>> 11:29:25:864 org.apache.zookeeper.server.quorum.LearnerHandler.run
>>>>> Follower sid: 2 : info : <ADDR>:2888:3888:participant
>>>>>
>>>>> 11:29:25:868 org.apache.zookeeper.server.ZKDatabase.isTxnLogSyncEnabled
>>>>> On disk txn sync enabled with snapshotSizeFactor 0.33
>>>>>
>>>>> 11:29:25:868
>>>>> org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
>>>>> Synchronizing with Learner sid: 2 maxCommittedLog=0x240025c7cc
>>>>> minCommittedLog=0x240025c5d0 lastProcessedZxid=0x240025c7cc
>>>>> peerLastZxid=0x240025c7c3
>>>>>
>>>>> 11:29:25:869
>>>>> org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
>>>>> Using committedLog for peer sid: 2
>>>>>
>>>>> 11:29:25:870
>>>>>
>> org.apache.zookeeper.server.quorum.LearnerHandler.queueCommittedProposals
>>>>> Sending DIFF zxid=0x240025c7cc for peer sid: 2
>>>>
>>>>
>>
>>
Re: Ephemeral Node Session ID Mismatch in ZooKeeper for a Rejoined Server
Posted by arjun s v <ar...@gmail.com>.
Andor,
Thank you for your response.
We are uncertain about the specific conditions triggering the issue, making
it difficult to predict its occurrence in version 3.8.4.
As multiple critical modules, including Hadoop and Kafka, rely on
Zookeeper, any disruption could lead to potential data loss on our end.
Identifying the exact root cause or scenario would help us both to either
fix or mitigate the issue.
I will provide any additional logs if needed.
Could you please clarify the specific issue and confirm if a fix is
available in version 3.8.4?
On Mon, Jun 23, 2025 at 9:27 PM Andor Molnar <an...@apache.org> wrote:
> Hi Arjun,
>
> Could you please validate the same scenario with latest stable version
> 3.8.4?
>
> Andor
>
>
>
>
> > On Jun 23, 2025, at 00:09, arjun s v <ar...@gmail.com> wrote:
> >
> > Continuation to the Ephemeral node issue,
> >
> > I observed that the learner sends ACKs for each packet it receives, but
> > there seems to be no verification on the leader's side to confirm these
> > ACKs against the packets sent.
> > Is there a configuration option that, when enabled, ensures all packet
> > ACKs, including COMMIT ACKs, are validated?
> > If packet loss is the reason for this issue, verifying all received ACKs
> > against the sent packets could help prevent such problems in the future.
> >
> > Please advise.
> >
> > On Thu, Jun 19, 2025 at 6:35 PM arjun s v <ar...@gmail.com> wrote:
> >
> >> Team,
> >>
> >> I'm investigating an issue where an ephemeral node in ZooKeeper was not
> >> properly managed after a server rejoined the ensemble. My setup uses
> >> ZooKeeper 3.9.3. Below is the timeline of events:
> >>
> >> Timeline:
> >>
> >> 1. An ephemeral node is created.
> >> 2. This is synced across all servers in the ensemble.
> >> 3. Follower 'A' goes out of the ensemble due to a connectivity issue.
> >> 4. Now the client session associated with the ephemeral node
> >> disconnects, deleting the ephemeral node across all active servers in
> the
> >> ensemble.
> >> 5. A new client session is initiated, creating another ephemeral node
> >> with the same path.
> >> 6. This new ephemeral node is synced across all active servers in the
> >> ensemble.
> >> 7. Follower 'A' rejoins the ensemble.
> >> 8. The leader syncs the latest commits to follower 'A'.
> >> 9. However, (Ephemeral Node).getEphemeralOwner() does not return the
> >> current session's session ID.
> >>
> >>
> >> I couldn't confirm if an old ephemeral node persisted, as the machine
> was
> >> restarted, resolving the issue. Debug logs were not enabled, so no
> >> additional logs are available to confirm the root cause. I suspect
> packet
> >> loss during the rejoin may have contributed. Attached are the
> >> leader-to-follower sync logs.
> >>
> >> Could you please advise if there are known issues with ephemeral node
> >> cleanup during server rejoins, or other scenarios to check? Is this
> likely
> >> due to packet loss or synchronization issues?
> >>
> >> Thanks in advance!
> >>
> >>
> >> Sync logs:
> >>
> >>> Follower
> >>>
> >>> 11:29:25:853
> >>> org.apache.zookeeper.server.quorum.Learner$LeaderConnector.run
> >>> Successfully connected to leader, using address: <DOMAIN/IPADDR>:2888
> >>>
> >>> 11:29:25:854 org.apache.zookeeper.util.SecurityUtils.createSaslClient
> >>> QuorumLearner will use DIGEST-MD5 as SASL mechanism.
> >>>
> >>> 11:29:25:859
> >>>
> org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.checkAuthStatus
> >>> Successfully completed the authentication using SASL. server addr:
> >>> <DOMAIN/IPADDR>:2888, status: SUCCESS
> >>>
> >>> 11:29:25:864 org.apache.zookeeper.server.quorum.QuorumPeer.setZabState
> >>> Peer state changed: following - synchronization
> >>>
> >>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> >>> Getting a diff from the leader 0x240025c7cc
> >>>
> >>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> >>> Got zxid 0x240025c7cc expected 0x1
> >>>
> >>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> >>> Learner received NEWLEADER message
> >>>
> >>> 11:29:25:869 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
> >>> Peer state changed: following - synchronization - diff
> >>>
> >>> 11:29:25:870
> >>> org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier
> >>> Dynamic reconfig is disabled, we don't store the last seen config.
> >>>
> >>> 11:29:25:871 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> >>> It took 2ms to persist and commit txns in packetsCommitted. 0
> outstanding
> >>> txns left in packetsNotLogged
> >>>
> >>> 11:29:25:873 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> >>> Set the current epoch to 37
> >>>
> >>> 11:29:25:874 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
> >>> Peer state changed: following - synchronization
> >>>
> >>> 11:29:25:874 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> >>> Sent NEWLEADER ack to leader with zxid 2500000000
> >>>
> >>> 11:29:25:879 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
> >>> Learner received UPTODATE message
> >>>
> >>>
> >>>
> >>> Leader:
> >>>
> >>> 11:29:25:864 org.apache.zookeeper.server.quorum.LearnerHandler.run
> >>> Follower sid: 2 : info : <ADDR>:2888:3888:participant
> >>>
> >>> 11:29:25:868 org.apache.zookeeper.server.ZKDatabase.isTxnLogSyncEnabled
> >>> On disk txn sync enabled with snapshotSizeFactor 0.33
> >>>
> >>> 11:29:25:868
> >>> org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
> >>> Synchronizing with Learner sid: 2 maxCommittedLog=0x240025c7cc
> >>> minCommittedLog=0x240025c5d0 lastProcessedZxid=0x240025c7cc
> >>> peerLastZxid=0x240025c7c3
> >>>
> >>> 11:29:25:869
> >>> org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
> >>> Using committedLog for peer sid: 2
> >>>
> >>> 11:29:25:870
> >>>
> org.apache.zookeeper.server.quorum.LearnerHandler.queueCommittedProposals
> >>> Sending DIFF zxid=0x240025c7cc for peer sid: 2
> >>
> >>
>
>
Re: Ephemeral Node Session ID Mismatch in ZooKeeper for a Rejoined Server
Posted by Andor Molnar <an...@apache.org>.
Hi Arjun,
Could you please validate the same scenario with latest stable version 3.8.4?
Andor
> On Jun 23, 2025, at 00:09, arjun s v <ar...@gmail.com> wrote:
>
> Continuation to the Ephemeral node issue,
>
> I observed that the learner sends ACKs for each packet it receives, but
> there seems to be no verification on the leader's side to confirm these
> ACKs against the packets sent.
> Is there a configuration option that, when enabled, ensures all packet
> ACKs, including COMMIT ACKs, are validated?
> If packet loss is the reason for this issue, verifying all received ACKs
> against the sent packets could help prevent such problems in the future.
>
> Please advise.
>
> On Thu, Jun 19, 2025 at 6:35 PM arjun s v <ar...@gmail.com> wrote:
>
>> Team,
>>
>> I'm investigating an issue where an ephemeral node in ZooKeeper was not
>> properly managed after a server rejoined the ensemble. My setup uses
>> ZooKeeper 3.9.3. Below is the timeline of events:
>>
>> Timeline:
>>
>> 1. An ephemeral node is created.
>> 2. This is synced across all servers in the ensemble.
>> 3. Follower 'A' goes out of the ensemble due to a connectivity issue.
>> 4. Now the client session associated with the ephemeral node
>> disconnects, deleting the ephemeral node across all active servers in the
>> ensemble.
>> 5. A new client session is initiated, creating another ephemeral node
>> with the same path.
>> 6. This new ephemeral node is synced across all active servers in the
>> ensemble.
>> 7. Follower 'A' rejoins the ensemble.
>> 8. The leader syncs the latest commits to follower 'A'.
>> 9. However, (Ephemeral Node).getEphemeralOwner() does not return the
>> current session's session ID.
>>
>>
>> I couldn't confirm if an old ephemeral node persisted, as the machine was
>> restarted, resolving the issue. Debug logs were not enabled, so no
>> additional logs are available to confirm the root cause. I suspect packet
>> loss during the rejoin may have contributed. Attached are the
>> leader-to-follower sync logs.
>>
>> Could you please advise if there are known issues with ephemeral node
>> cleanup during server rejoins, or other scenarios to check? Is this likely
>> due to packet loss or synchronization issues?
>>
>> Thanks in advance!
>>
>>
>> Sync logs:
>>
>>> Follower
>>>
>>> 11:29:25:853
>>> org.apache.zookeeper.server.quorum.Learner$LeaderConnector.run
>>> Successfully connected to leader, using address: <DOMAIN/IPADDR>:2888
>>>
>>> 11:29:25:854 org.apache.zookeeper.util.SecurityUtils.createSaslClient
>>> QuorumLearner will use DIGEST-MD5 as SASL mechanism.
>>>
>>> 11:29:25:859
>>> org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.checkAuthStatus
>>> Successfully completed the authentication using SASL. server addr:
>>> <DOMAIN/IPADDR>:2888, status: SUCCESS
>>>
>>> 11:29:25:864 org.apache.zookeeper.server.quorum.QuorumPeer.setZabState
>>> Peer state changed: following - synchronization
>>>
>>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>> Getting a diff from the leader 0x240025c7cc
>>>
>>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>> Got zxid 0x240025c7cc expected 0x1
>>>
>>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>> Learner received NEWLEADER message
>>>
>>> 11:29:25:869 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
>>> Peer state changed: following - synchronization - diff
>>>
>>> 11:29:25:870
>>> org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier
>>> Dynamic reconfig is disabled, we don't store the last seen config.
>>>
>>> 11:29:25:871 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>> It took 2ms to persist and commit txns in packetsCommitted. 0 outstanding
>>> txns left in packetsNotLogged
>>>
>>> 11:29:25:873 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>> Set the current epoch to 37
>>>
>>> 11:29:25:874 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
>>> Peer state changed: following - synchronization
>>>
>>> 11:29:25:874 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>> Sent NEWLEADER ack to leader with zxid 2500000000
>>>
>>> 11:29:25:879 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>>> Learner received UPTODATE message
>>>
>>>
>>>
>>> Leader:
>>>
>>> 11:29:25:864 org.apache.zookeeper.server.quorum.LearnerHandler.run
>>> Follower sid: 2 : info : <ADDR>:2888:3888:participant
>>>
>>> 11:29:25:868 org.apache.zookeeper.server.ZKDatabase.isTxnLogSyncEnabled
>>> On disk txn sync enabled with snapshotSizeFactor 0.33
>>>
>>> 11:29:25:868
>>> org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
>>> Synchronizing with Learner sid: 2 maxCommittedLog=0x240025c7cc
>>> minCommittedLog=0x240025c5d0 lastProcessedZxid=0x240025c7cc
>>> peerLastZxid=0x240025c7c3
>>>
>>> 11:29:25:869
>>> org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
>>> Using committedLog for peer sid: 2
>>>
>>> 11:29:25:870
>>> org.apache.zookeeper.server.quorum.LearnerHandler.queueCommittedProposals
>>> Sending DIFF zxid=0x240025c7cc for peer sid: 2
>>
>>
Re: Ephemeral Node Session ID Mismatch in ZooKeeper for a Rejoined Server
Posted by arjun s v <ar...@gmail.com>.
Continuation to the Ephemeral node issue,
I observed that the learner sends ACKs for each packet it receives, but
there seems to be no verification on the leader's side to confirm these
ACKs against the packets sent.
Is there a configuration option that, when enabled, ensures all packet
ACKs, including COMMIT ACKs, are validated?
If packet loss is the reason for this issue, verifying all received ACKs
against the sent packets could help prevent such problems in the future.
Please advise.
On Thu, Jun 19, 2025 at 6:35 PM arjun s v <ar...@gmail.com> wrote:
> Team,
>
> I'm investigating an issue where an ephemeral node in ZooKeeper was not
> properly managed after a server rejoined the ensemble. My setup uses
> ZooKeeper 3.9.3. Below is the timeline of events:
>
> Timeline:
>
> 1. An ephemeral node is created.
> 2. This is synced across all servers in the ensemble.
> 3. Follower 'A' goes out of the ensemble due to a connectivity issue.
> 4. Now the client session associated with the ephemeral node
> disconnects, deleting the ephemeral node across all active servers in the
> ensemble.
> 5. A new client session is initiated, creating another ephemeral node
> with the same path.
> 6. This new ephemeral node is synced across all active servers in the
> ensemble.
> 7. Follower 'A' rejoins the ensemble.
> 8. The leader syncs the latest commits to follower 'A'.
> 9. However, (Ephemeral Node).getEphemeralOwner() does not return the
> current session's session ID.
>
>
> I couldn't confirm if an old ephemeral node persisted, as the machine was
> restarted, resolving the issue. Debug logs were not enabled, so no
> additional logs are available to confirm the root cause. I suspect packet
> loss during the rejoin may have contributed. Attached are the
> leader-to-follower sync logs.
>
> Could you please advise if there are known issues with ephemeral node
> cleanup during server rejoins, or other scenarios to check? Is this likely
> due to packet loss or synchronization issues?
>
> Thanks in advance!
>
>
> Sync logs:
>
>> Follower
>>
>> 11:29:25:853
>> org.apache.zookeeper.server.quorum.Learner$LeaderConnector.run
>> Successfully connected to leader, using address: <DOMAIN/IPADDR>:2888
>>
>> 11:29:25:854 org.apache.zookeeper.util.SecurityUtils.createSaslClient
>> QuorumLearner will use DIGEST-MD5 as SASL mechanism.
>>
>> 11:29:25:859
>> org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthLearner.checkAuthStatus
>> Successfully completed the authentication using SASL. server addr:
>> <DOMAIN/IPADDR>:2888, status: SUCCESS
>>
>> 11:29:25:864 org.apache.zookeeper.server.quorum.QuorumPeer.setZabState
>> Peer state changed: following - synchronization
>>
>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> Getting a diff from the leader 0x240025c7cc
>>
>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> Got zxid 0x240025c7cc expected 0x1
>>
>> 11:29:25:869 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> Learner received NEWLEADER message
>>
>> 11:29:25:869 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
>> Peer state changed: following - synchronization - diff
>>
>> 11:29:25:870
>> org.apache.zookeeper.server.quorum.QuorumPeer.setLastSeenQuorumVerifier
>> Dynamic reconfig is disabled, we don't store the last seen config.
>>
>> 11:29:25:871 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> It took 2ms to persist and commit txns in packetsCommitted. 0 outstanding
>> txns left in packetsNotLogged
>>
>> 11:29:25:873 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> Set the current epoch to 37
>>
>> 11:29:25:874 org.apache.zookeeper.server.quorum.QuorumPeer.setSyncMode
>> Peer state changed: following - synchronization
>>
>> 11:29:25:874 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> Sent NEWLEADER ack to leader with zxid 2500000000
>>
>> 11:29:25:879 org.apache.zookeeper.server.quorum.Learner.syncWithLeader
>> Learner received UPTODATE message
>>
>>
>>
>> Leader:
>>
>> 11:29:25:864 org.apache.zookeeper.server.quorum.LearnerHandler.run
>> Follower sid: 2 : info : <ADDR>:2888:3888:participant
>>
>> 11:29:25:868 org.apache.zookeeper.server.ZKDatabase.isTxnLogSyncEnabled
>> On disk txn sync enabled with snapshotSizeFactor 0.33
>>
>> 11:29:25:868
>> org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
>> Synchronizing with Learner sid: 2 maxCommittedLog=0x240025c7cc
>> minCommittedLog=0x240025c5d0 lastProcessedZxid=0x240025c7cc
>> peerLastZxid=0x240025c7c3
>>
>> 11:29:25:869
>> org.apache.zookeeper.server.quorum.LearnerHandler.syncFollower
>> Using committedLog for peer sid: 2
>>
>> 11:29:25:870
>> org.apache.zookeeper.server.quorum.LearnerHandler.queueCommittedProposals
>> Sending DIFF zxid=0x240025c7cc for peer sid: 2
>
>