ZOOKEEPER-2024: Boosting ZooKeeper throughput 10x

[Alexander Shraer <sh...@gmail.com>]

Hi All,

Kfir completed the work on ZK-2024 and I'd like to commit it, ideally
targeting 3.5.2. It would be great if others could review it too. The
actual code change in this JIRA is small, most of it are new tests.

It addresses a well known issue in ZooKeeper: writers block readers simply
because they are connected to the same server, something that is not
mandated by ZK semantics. The main idea of the fix is simple -- writes
should only block operations of the same client. The patch also solves a
starvation bug in the commit processor.

This brings up to x10 times throughput boost in saturated mixed workloads
(where readers and writers connect to the same ZooKeeper server). The gains
can even be larger (we saw up to 17x) with non-saturated workloads. In
particular, we saw these gains also for cross-dc deployments since
Observers run the commit processor too.

The JIRA includes a short document <https://goo.gl/m1cINJ> that explains
the new algorithm and its correctness. And a second document
<https://goo.gl/W0xDUP> that shows the performance gains with various
workloads.

Cheers,
Alex