Fwd: How Spark coordinates multi contender race on writing zookeeper? (Also on stackoverflow)

[Zili Chen <wa...@gmail.com>]

Hi guys,

Recently I open a question[1] on StackOverflow about leader election
with ZooKeeper high-availability backend. It puzzles me for some days
and it would be really help if you can take a look or even give some
thoughts.

Copy the content to mailing list:

Spark uses Curator#LeaderLatch for leader election. And PersistenceEngine
for persistent. I'd like to know what if an old leader lost its leadership
but before got notified, a new leader started to work, in which case both
of the two contenders regarded themselves as the leader and wrote on
zookeeper. Isn't it a race condition if the new leader just create a znode
on zookeeper but the old one, which was no longer the leader, delete it.

Best,
tison.

[1]
https://stackoverflow.com/questions/55380498/how-spark-coordinates-multi-contender-race-on-writing-zookeeper