ResourceManager HA + ZooKeeper performance questions

[Ben Podgursky <bp...@gmail.com>]

Hi all,

I'm looking into turning on High Availability ResourceManagers in our YARN
cluster (we run a single ResourceManager at the moment).  We plan on using
ZooKeeper for fencing and state storage, and my main concern is around ZK
scalability+performance.  We run a fairly active YARN cluster:

- ~550 NodeManagers
- ~5,000 applications/day
- ~15,000 active containers at any given time
~ usually ~100 applications running at any given time

Since we can't really load-test our setup before turning HA on in
production, I was hoping someone who had run a cluster at similar scale
could give advice on their ZK environment; specifically

- What ZK heap size did you need?
- How many nodes in your ensemble?
- What kind of disks?  Are spinning disks OK, or do you use SSDs?
- Did you need any special configurations around timeouts, etc?

Basically I'm looking either for any horror stories, or hoping that someone
can say that RM HA will be A-OK at this throughput.


Thanks,

Ben