You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@curator.apache.org by Jordan Zimmerman <jo...@jordanzimmerman.com> on 2021/11/05 12:40:47 UTC
Re: [External Sender] Double Leadership Issue
I wonder if we should add a watcher to the ZNode created by the LeaderLatch. That way, we can handle the case where the ZNode disappears underneath the latch. Other than that, it would be very helpful to have a test case for this. Can you try to create one please? My involvement with the project is very limited these days but if there's a good test case I can find some time to make changes.
>> Possible Solution (where we would like to hear your thoughts/suggestions):
>>
>> * The current curator code during reset() does
>> * setLeadership(false) first followed by
>> * setNode(null) (i.e. deleting its latch node)
>>
>> * Swapping these two should resolve the issue, as we setting leadership to false once after its latch node gets deleted
>> * setNode(null) (i.e. deleting its latch node) first followed by
>> * setLeadership(false)
The problem here is that deletion isn't guaranteed here. It could happen at a later time when there are network partitions.
-Jordan
> On Nov 3, 2021, at 3:33 PM, Viswanathan Rajagopal <vi...@workday.com.INVALID> wrote:
>
> Hi Jordan,
>
> The dual leadership continue indefinitely in my case
>
> Many Thanks,
> Viswa
>
> From: Jordan Zimmerman <jo...@jordanzimmerman.com>
> Date: Wednesday, 3 November 2021 at 08:02
> To: dev@curator.apache.org <de...@curator.apache.org>
> Cc: user@curator.apache.org <us...@curator.apache.org>
> Subject: [External Sender] Re: Double Leadership Issue
> Do I understand this correctly that there are two leaders for a short period of time - i.e. it corrects itself eventually? Or does the dual leadership continue indefinitely?
>
> -Jordan
>
>> On Nov 2, 2021, at 11:48 AM, Viswanathan Rajagopal <vi...@workday.com.INVALID> wrote:
>>
>> Hello Team,
>>
>> Greetings!
>> Any update on the below mentioned observation?
>>
>> Many Thanks,
>> Viswa
>>
>> From: Viswanathan Rajagopal <vi...@workday.com.INVALID>
>> Date: Wednesday, 27 October 2021 at 16:15
>> To: dev@curator.apache.org <de...@curator.apache.org>, user@curator.apache.org <us...@curator.apache.org>
>> Subject: [External Sender] Double Leadership Issue
>> Hello Team,
>>
>> Greetings!
>> While using Curator Leader Latch Recipe in our application, we observed a potential issue where two clients have become a leader. Raised a Jira on the same for your reference (Jira Link : https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CURATOR-2D620&d=DwIF-g&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjU&m=nFB4puWyvVe8eRiZ3oi_C8Ao1WkqCb9wsonPrIl3LY8&s=3LDys_XJLYEnQ0_K3auTUo8DsOom0xZAMAC7ASgkt0A&e= )
>> Quick summary of below description
>>
>> * Our use case explained
>> * Issue details
>> * Timeline of events mentioned
>> * Attached test code to reproduce the reported issue
>> * Possible solution given, where we need your suggestions
>> Our use case:
>>
>> * Two clients trying to get the leadership using Curator Leader Latch Recipe. On LeaderLatchListener.isLeader() Client would become a leader and on LeaderLatchListener.notLeader() Client would lose its leadership
>> Issue details:
>>
>> * One of the clients on receiving two CuratorConnectionListener RECONNECTED events in quick succession, we observed that LeaderLatch EventThreads interleave with each other, resulting in "latch node deletion" happen after "client becoming a leader", thereby the client will still be a leader though its corresponding latch node has been deleted
>> * And the other client who tried to get leadership creates its latch node and sees itself in first index and thus become a leader
>> * So at this point, two clients have become a leader
>>
>> Timeline of events:
>>
>> * Timeline events of Client A whose corresponding latch node is deleted but still be a leader
>> * At t1, 1st RECONNECTED event fired
>> * At t2, [ EventThread of 1st RECONNECTED event ] Resets leadership (true -> false)
>> * At t3, [ EventThread of 1st RECONNECTED event ] Fire “listener.notLeader()”
>> * At t4, [ EventThread of 1st RECONNECTED event ] Deletes latch node
>> * At t5, [ EventThread of 1st RECONNECTED event ] Creates new latch node
>> * At t6, 2nd RECONNECTED event fired
>> * At t7, [ EventThread of 2nd RECONNECTED event ] Resets leadership (false -> false), Basically NOP
>> * At t8, [ EventThread of 2nd RECONNECTED event ] Fire nothing. Basically NOP
>> * At t9, [ EventThread of 1st RECONNECTED event ] Get children -> sort them -> check leadership -> Set leadership to true -> Fire “Has become a leader” leader listener event
>> * At t10, [ EventThread of 2nd RECONNECTED event ] Delete latch node (which actually deletes the latch node with which the Client A has become a leader through previous step)
>>
>> * Timeline events of Client B who also become a leader
>> * At t11, Client B creates its latch node -> Get children -> sort them -> check leadership -> Set leadership to true -> Fire “Has become a leader” leader listener event
>>
>> This ends up in a situation where both Client A and Client B have become a leader
>>
>> As we observe, over the period (t8 -> t10), Client A’s LeaderLatch EventThreads interleave with each other causing leadership latch node deleted but local state still assumes that it’s a leader
>>
>> Reproducing the issue:
>>
>> * Wrote a Junit test case firing an artificial curator connection reconnected events and simulated LeaderLatch EventThreads interleave through CountDownLatches
>> * Test simulator for 2.5.0:
>> * https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_6a78a3a0de032212175d80caa64f140c743219ae&d=DwIF-g&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjU&m=nFB4puWyvVe8eRiZ3oi_C8Ao1WkqCb9wsonPrIl3LY8&s=tveG7d6kAd8SeywmuCN7zyd1ufTvARJdEEc0gxTs2rU&e=
>> * https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_d2b1b33a6885c05619c058aa2bee63962fd6fa08&d=DwIF-g&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjU&m=nFB4puWyvVe8eRiZ3oi_C8Ao1WkqCb9wsonPrIl3LY8&s=jixCmfLZiaseXsSWihiUiYMw8cj5cDg1O6gLFJY3kKg&e=
>> * Test Simulator for latest Curator version:
>> * https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_0949137f7323a1d5f34afc85a7042e8d9e85a8bc&d=DwIF-g&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjU&m=nFB4puWyvVe8eRiZ3oi_C8Ao1WkqCb9wsonPrIl3LY8&s=bzLny0aqbqUHmvLwkWyLdIySm65swqv2rAT1Kn0MKJ0&e=
>> * https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_1aadd4b5dbc8811a2e7a49b92f29170333e8ba4a&d=DwIF-g&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjU&m=nFB4puWyvVe8eRiZ3oi_C8Ao1WkqCb9wsonPrIl3LY8&s=GTlqqRRRB_P5y_f1tRSRxv1HZvVjhwFHtlogEk47LAU&e=
>>
>> Possible Solution (where we would like to hear your thoughts/suggestions):
>>
>> * The current curator code during reset() does
>> * setLeadership(false) first followed by
>> * setNode(null) (i.e. deleting its latch node)
>>
>> * Swapping these two should resolve the issue, as we setting leadership to false once after its latch node gets deleted
>> * setNode(null) (i.e. deleting its latch node) first followed by
>> * setLeadership(false)
>>
>> Many Thanks,
>> Viswa
Re: [External Sender] Double Leadership Issue
Posted by Viswanathan Rajagopal <vi...@workday.com.INVALID>.
Sure will work on that and share it
Many Thanks,
Viswa
From: Jordan Zimmerman <jo...@jordanzimmerman.com>
Date: Monday, 8 November 2021 at 10:50
To: dev@curator.apache.org <de...@curator.apache.org>
Cc: user@curator.apache.org <us...@curator.apache.org>
Subject: Re: [External Sender] Double Leadership Issue
It would be better if these test cases are turned into unit tests so we can have them in the code base to prevent future problems. Can you please turn them into Unit Tests?
-Jordan
> On Nov 6, 2021, at 8:52 PM, Viswanathan Rajagopal <vi...@workday.com.INVALID> wrote:
>
> Have linked the test cases in my original mail.
> Test cases below,
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_0949137f7323a1d5f34afc85a7042e8d9e85a8bc&d=DwIFAg&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjVR9Zfxevy2QbIqXzpZz32m&m=ECHFKudcTGfT3E82lmLs7roaeMuzE_iCVJNGfwalltDWA9lr0GCgOeGBj-TBFAK-&s=SU_e5NGXgyaxzb45B-FzGoJSASdDqsvBFpf2AxFqd_E&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_0949137f7323a1d5f34afc85a7042e8d9e85a8bc&d=DwIFAg&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjVR9Zfxevy2QbIqXzpZz32m&m=ECHFKudcTGfT3E82lmLs7roaeMuzE_iCVJNGfwalltDWA9lr0GCgOeGBj-TBFAK-&s=SU_e5NGXgyaxzb45B-FzGoJSASdDqsvBFpf2AxFqd_E&e= >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_1aadd4b5dbc8811a2e7a49b92f29170333e8ba4a&d=DwIFAg&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjVR9Zfxevy2QbIqXzpZz32m&m=ECHFKudcTGfT3E82lmLs7roaeMuzE_iCVJNGfwalltDWA9lr0GCgOeGBj-TBFAK-&s=dgwkXAmh1QnRpbPvjkH6uAJ3rcFptph6SCVhe1nPM3A&e= <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_1aadd4b5dbc8811a2e7a49b92f29170333e8ba4a&d=DwIFAg&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjVR9Zfxevy2QbIqXzpZz32m&m=ECHFKudcTGfT3E82lmLs7roaeMuzE_iCVJNGfwalltDWA9lr0GCgOeGBj-TBFAK-&s=dgwkXAmh1QnRpbPvjkH6uAJ3rcFptph6SCVhe1nPM3A&e= >
>
Re: [External Sender] Double Leadership Issue
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
It would be better if these test cases are turned into unit tests so we can have them in the code base to prevent future problems. Can you please turn them into Unit Tests?
-Jordan
> On Nov 6, 2021, at 8:52 PM, Viswanathan Rajagopal <vi...@workday.com.INVALID> wrote:
>
> Have linked the test cases in my original mail.
> Test cases below,
> https://github.com/ViswaNXplore/curator/commit/0949137f7323a1d5f34afc85a7042e8d9e85a8bc <https://github.com/ViswaNXplore/curator/commit/0949137f7323a1d5f34afc85a7042e8d9e85a8bc>
> https://github.com/ViswaNXplore/curator/commit/1aadd4b5dbc8811a2e7a49b92f29170333e8ba4a <https://github.com/ViswaNXplore/curator/commit/1aadd4b5dbc8811a2e7a49b92f29170333e8ba4a>
>
Re: [External Sender] Double Leadership Issue
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
It would be better if these test cases are turned into unit tests so we can have them in the code base to prevent future problems. Can you please turn them into Unit Tests?
-Jordan
> On Nov 6, 2021, at 8:52 PM, Viswanathan Rajagopal <vi...@workday.com.INVALID> wrote:
>
> Have linked the test cases in my original mail.
> Test cases below,
> https://github.com/ViswaNXplore/curator/commit/0949137f7323a1d5f34afc85a7042e8d9e85a8bc <https://github.com/ViswaNXplore/curator/commit/0949137f7323a1d5f34afc85a7042e8d9e85a8bc>
> https://github.com/ViswaNXplore/curator/commit/1aadd4b5dbc8811a2e7a49b92f29170333e8ba4a <https://github.com/ViswaNXplore/curator/commit/1aadd4b5dbc8811a2e7a49b92f29170333e8ba4a>
>
Re: [External Sender] Double Leadership Issue
Posted by Viswanathan Rajagopal <vi...@workday.com.INVALID>.
>> I wonder if we should add a watcher to the ZNode created by the LeaderLatch. That way, we can handle the case where the ZNode disappears underneath the latch.
Nice suggestion. Should work. The other quick solution that just strikes me is -
Don’t “reset leadership” when its previous “reset leadership” in progress.
Doing “reset leadership” when its previous “reset leadership” in progress is what causes this interleaving and thereby dual leadership.
So Protecting Resetting process by defining a synchronised local Boolean variable (isResetting) should avoid this issue
* At start of resetting
* If (isResetting), then do nothing
* Else set isResetting to true and start resetting leadership process
* At the end of resetting, set isResetting to false
*
>> Other than that, it would be very helpful to have a test case for this.
Have linked the test cases in my original mail.
Test cases below,
https://github.com/ViswaNXplore/curator/commit/0949137f7323a1d5f34afc85a7042e8d9e85a8bc
https://github.com/ViswaNXplore/curator/commit/1aadd4b5dbc8811a2e7a49b92f29170333e8ba4a
Many Thanks,
Viswa
From: Jordan Zimmerman <jo...@jordanzimmerman.com>
Date: Friday, 5 November 2021 at 12:42
To: dev@curator.apache.org <de...@curator.apache.org>
Cc: user@curator.apache.org <us...@curator.apache.org>
Subject: Re: [External Sender] Double Leadership Issue
I wonder if we should add a watcher to the ZNode created by the LeaderLatch. That way, we can handle the case where the ZNode disappears underneath the latch. Other than that, it would be very helpful to have a test case for this. Can you try to create one please? My involvement with the project is very limited these days but if there's a good test case I can find some time to make changes.
>> Possible Solution (where we would like to hear your thoughts/suggestions):
>>
>> * The current curator code during reset() does
>> * setLeadership(false) first followed by
>> * setNode(null) (i.e. deleting its latch node)
>>
>> * Swapping these two should resolve the issue, as we setting leadership to false once after its latch node gets deleted
>> * setNode(null) (i.e. deleting its latch node) first followed by
>> * setLeadership(false)
The problem here is that deletion isn't guaranteed here. It could happen at a later time when there are network partitions.
-Jordan
> On Nov 3, 2021, at 3:33 PM, Viswanathan Rajagopal <vi...@workday.com.INVALID> wrote:
>
> Hi Jordan,
>
> The dual leadership continue indefinitely in my case
>
> Many Thanks,
> Viswa
>
> From: Jordan Zimmerman <jo...@jordanzimmerman.com>
> Date: Wednesday, 3 November 2021 at 08:02
> To: dev@curator.apache.org <de...@curator.apache.org>
> Cc: user@curator.apache.org <us...@curator.apache.org>
> Subject: [External Sender] Re: Double Leadership Issue
> Do I understand this correctly that there are two leaders for a short period of time - i.e. it corrects itself eventually? Or does the dual leadership continue indefinitely?
>
> -Jordan
>
>> On Nov 2, 2021, at 11:48 AM, Viswanathan Rajagopal <vi...@workday.com.INVALID> wrote:
>>
>> Hello Team,
>>
>> Greetings!
>> Any update on the below mentioned observation?
>>
>> Many Thanks,
>> Viswa
>>
>> From: Viswanathan Rajagopal <vi...@workday.com.INVALID>
>> Date: Wednesday, 27 October 2021 at 16:15
>> To: dev@curator.apache.org <de...@curator.apache.org>, user@curator.apache.org <us...@curator.apache.org>
>> Subject: [External Sender] Double Leadership Issue
>> Hello Team,
>>
>> Greetings!
>> While using Curator Leader Latch Recipe in our application, we observed a potential issue where two clients have become a leader. Raised a Jira on the same for your reference (Jira Link : https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CURATOR-2D620&d=DwIF-g&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjU&m=nFB4puWyvVe8eRiZ3oi_C8Ao1WkqCb9wsonPrIl3LY8&s=3LDys_XJLYEnQ0_K3auTUo8DsOom0xZAMAC7ASgkt0A&e= )
>> Quick summary of below description
>>
>> * Our use case explained
>> * Issue details
>> * Timeline of events mentioned
>> * Attached test code to reproduce the reported issue
>> * Possible solution given, where we need your suggestions
>> Our use case:
>>
>> * Two clients trying to get the leadership using Curator Leader Latch Recipe. On LeaderLatchListener.isLeader() Client would become a leader and on LeaderLatchListener.notLeader() Client would lose its leadership
>> Issue details:
>>
>> * One of the clients on receiving two CuratorConnectionListener RECONNECTED events in quick succession, we observed that LeaderLatch EventThreads interleave with each other, resulting in "latch node deletion" happen after "client becoming a leader", thereby the client will still be a leader though its corresponding latch node has been deleted
>> * And the other client who tried to get leadership creates its latch node and sees itself in first index and thus become a leader
>> * So at this point, two clients have become a leader
>>
>> Timeline of events:
>>
>> * Timeline events of Client A whose corresponding latch node is deleted but still be a leader
>> * At t1, 1st RECONNECTED event fired
>> * At t2, [ EventThread of 1st RECONNECTED event ] Resets leadership (true -> false)
>> * At t3, [ EventThread of 1st RECONNECTED event ] Fire “listener.notLeader()”
>> * At t4, [ EventThread of 1st RECONNECTED event ] Deletes latch node
>> * At t5, [ EventThread of 1st RECONNECTED event ] Creates new latch node
>> * At t6, 2nd RECONNECTED event fired
>> * At t7, [ EventThread of 2nd RECONNECTED event ] Resets leadership (false -> false), Basically NOP
>> * At t8, [ EventThread of 2nd RECONNECTED event ] Fire nothing. Basically NOP
>> * At t9, [ EventThread of 1st RECONNECTED event ] Get children -> sort them -> check leadership -> Set leadership to true -> Fire “Has become a leader” leader listener event
>> * At t10, [ EventThread of 2nd RECONNECTED event ] Delete latch node (which actually deletes the latch node with which the Client A has become a leader through previous step)
>>
>> * Timeline events of Client B who also become a leader
>> * At t11, Client B creates its latch node -> Get children -> sort them -> check leadership -> Set leadership to true -> Fire “Has become a leader” leader listener event
>>
>> This ends up in a situation where both Client A and Client B have become a leader
>>
>> As we observe, over the period (t8 -> t10), Client A’s LeaderLatch EventThreads interleave with each other causing leadership latch node deleted but local state still assumes that it’s a leader
>>
>> Reproducing the issue:
>>
>> * Wrote a Junit test case firing an artificial curator connection reconnected events and simulated LeaderLatch EventThreads interleave through CountDownLatches
>> * Test simulator for 2.5.0:
>> * https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_6a78a3a0de032212175d80caa64f140c743219ae&d=DwIF-g&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjU&m=nFB4puWyvVe8eRiZ3oi_C8Ao1WkqCb9wsonPrIl3LY8&s=tveG7d6kAd8SeywmuCN7zyd1ufTvARJdEEc0gxTs2rU&e=
>> * https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_d2b1b33a6885c05619c058aa2bee63962fd6fa08&d=DwIF-g&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjU&m=nFB4puWyvVe8eRiZ3oi_C8Ao1WkqCb9wsonPrIl3LY8&s=jixCmfLZiaseXsSWihiUiYMw8cj5cDg1O6gLFJY3kKg&e=
>> * Test Simulator for latest Curator version:
>> * https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_0949137f7323a1d5f34afc85a7042e8d9e85a8bc&d=DwIF-g&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjU&m=nFB4puWyvVe8eRiZ3oi_C8Ao1WkqCb9wsonPrIl3LY8&s=bzLny0aqbqUHmvLwkWyLdIySm65swqv2rAT1Kn0MKJ0&e=
>> * https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ViswaNXplore_curator_commit_1aadd4b5dbc8811a2e7a49b92f29170333e8ba4a&d=DwIF-g&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=mwSLlPO0Vtstmu1dce0TMFqf5lUxD2SPdNdc1k4NXjU&m=nFB4puWyvVe8eRiZ3oi_C8Ao1WkqCb9wsonPrIl3LY8&s=GTlqqRRRB_P5y_f1tRSRxv1HZvVjhwFHtlogEk47LAU&e=
>>
>> Possible Solution (where we would like to hear your thoughts/suggestions):
>>
>> * The current curator code during reset() does
>> * setLeadership(false) first followed by
>> * setNode(null) (i.e. deleting its latch node)
>>
>> * Swapping these two should resolve the issue, as we setting leadership to false once after its latch node gets deleted
>> * setNode(null) (i.e. deleting its latch node) first followed by
>> * setLeadership(false)
>>
>> Many Thanks,
>> Viswa