You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@curator.apache.org by "jira-importer (via GitHub)" <gi...@apache.org> on 2025/01/21 13:12:44 UTC
[I] [CURATOR-62] Leader Election Deadlock [curator]
jira-importer opened a new issue, #583:
URL: https://github.com/apache/curator/issues/583
<p>I've noticed that it is possible for a leader election to deadlock if a thread is interrupted while it is trying to acquire the mutex for the election.</p>
<p>I've created a forced example of this here: <a href="https://github.com/dfjones/curator/commit/544220b1e6b51c2718a7d3511a74962ff1c5ff48" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/dfjones/curator/commit/544220b1e6b51c2718a7d3511a74962ff1c5ff48</a></p>
<p>You can see deadlock by using my modified code and running the LeaderSelectorExample. Some leaders may execute, but on my system I eventually see deadlock. Note that I only see deadlock when running against a remote zk server rather than the embedded test server. I'm using Zookeeper 3.4.5 on Mac OS X 10.8.4.</p>
<p>From what I can tell by inspecting the ZK state/watching in the debugger, the thread that is interrupted is able to successfully create the lock object in ZK. However, due to the interrupt an exception is generated and LockInternals#internalLockLoop never runs. Later, in LeaderSelector#doWork when mutex.release() is called this fails at the for lockData.</p>
<p>Once this occurs, the lock object in ZK is the oldest and will cause deadlock.</p>
---
<details><summary><i>Originally reported by <a title="Duncan Jones" href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=djones">djones</a>, imported from: <a href="https://issues.apache.org/jira/browse/CURATOR-62" target="_blank">Leader Election Deadlock</a></i></summary>
<i><ul>
<li><b>assignee</b>: <a title="Jordan Zimmerman" href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>
<li><b>status</b>: Open
<li><b>priority</b>: Minor
<li><b>resolution</b>: Unresolved
<li><b>imported</b>: 2025-01-21
</ul></i>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [CURATOR-62] Leader Election Deadlock [curator]
Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #583:
URL: https://github.com/apache/curator/issues/583#issuecomment-2604700204
<i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=djones">djones</a>:</i>
<p>I was able to retest this and it appears to be fixed on master.</p>
<p>I created a better test at the version where I first noticed this bug:<br/>
<a href="https://github.com/dfjones/curator/tree/CURATOR-62" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/dfjones/curator/tree/CURATOR-62</a></p>
<p>This test fails 100% of the time for me. (It might be a good test to include, but I had to modify LeaderSelector to get the interrupt to occur at the exact time to cause the issue.)</p>
<p>I ported this test to the latest code on master here:<br/>
<a href="https://github.com/dfjones/curator/tree/CURATOR-62-Latest" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/dfjones/curator/tree/CURATOR-62-Latest</a></p>
<p>This test passes 100% of the time for me. I believe this issue can be closed. It appears the fix for <a href="https://issues.apache.org/jira/browse/CURATOR-202" title="LeaderSelector node is not removed on connection loss if reconnected to the same session" class="issue-link" data-issue-key="CURATOR-202"><del>CURATOR-202</del></a> also fixes this issue.</p>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [CURATOR-62] Leader Election Deadlock [curator]
Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #583:
URL: https://github.com/apache/curator/issues/583#issuecomment-2604700193
<i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>:</i>
<p>I've done some checking and this is separate from <a href="https://issues.apache.org/jira/browse/CURATOR-45" title="LeaderSelector threw exception, but still created ephemeral node, breaking everything" class="issue-link" data-issue-key="CURATOR-45"><del>CURATOR-45</del></a>. </p>
<p>I believe that if a thread is being interrupted then the Curator instance is most likely being closed. So, I don't think this will happen in practice. But, we should fix anyway.</p>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [CURATOR-62] Leader Election Deadlock [curator]
Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #583:
URL: https://github.com/apache/curator/issues/583#issuecomment-2604700196
<i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=djones">djones</a>:</i>
<p>I came across this bug because I've run into it in production. Perhaps I have a weird use case, but what I'm essentially doing is running an election on a fixed schedule (say every 1 hour). When its work is finished, the leader signals for shutdown and then exits. There's a race condition between another thread trying to become the leader and handling the shutdown event that can result in the deadlock described here for future elections. This wouldn't be an issue if leaderSelector#close was completely reliable.</p>
<p>I can probably work around this bug, but it's definitely an issue for repeated elections.</p>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [CURATOR-62] Leader Election Deadlock [curator]
Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #583:
URL: https://github.com/apache/curator/issues/583#issuecomment-2604700197
<i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>:</i>
<p>This might be a dup of <a href="https://issues.apache.org/jira/browse/CURATOR-202" title="LeaderSelector node is not removed on connection loss if reconnected to the same session" class="issue-link" data-issue-key="CURATOR-202"><del>CURATOR-202</del></a> - please retest and report back</p>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [CURATOR-62] Leader Election Deadlock [curator]
Posted by "jira-importer (via GitHub)" <gi...@apache.org>.
jira-importer commented on issue #583:
URL: https://github.com/apache/curator/issues/583#issuecomment-2604700187
<i><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=randgalt">randgalt</a>:</i>
<p>This might be the same as <a href="https://issues.apache.org/jira/browse/CURATOR-45" title="LeaderSelector threw exception, but still created ephemeral node, breaking everything" class="issue-link" data-issue-key="CURATOR-45"><del>CURATOR-45</del></a></p>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@curator.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org