You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Amar Gajbhiye <Am...@1eQ.com> on 2017/02/07 07:24:20 UTC
Extremely different readings on different zookeeper deployments
Hi,
I am working on a distributed system where I want to use consensus algorithm. I am evaluating ZooKeeper for the same.
I am trying to quantify read and write operations for zookeeper for version 3.5.2.
I am running following operations.
1. Create 50k Znodes with data
2. Read from those 50k znodes
I tried this operations when zookeeper server was running on 4 different remote servers in standalone mode. My client was always on same machine.
1. Test case 1
Server configuration : Windows 8 server R2.
jdk 1.7.0_80
Avg time required for each write operations : 1.8 ms
Avg time required for each read operations : 0.9 ms
2. Test case 2
Server configuration : Windows 8 server R2.
jdk 1.7.0_21
Avg time required for each write operation : 1.7 ms
Avg time required for each read operation : 0.9 ms
3. Test case 3
Server configuration : Windows 7
jdk 1.7.0_25
Avg time required for each write operation : 35 ms
Avg time required for each read operation : 0.8 ms
4. Test case 4
Server configuration : Windows 7
jdk1.8.0_101
Avg time required for each write operation : 40 ms
Avg time required for each read operation : 0.9 ms
5. To get network latency out of picture, I kept both client and server on same machine with following configuration
Server configuration : Windows 7
jdk1.8.0_101
Avg time required for each write operation : 30 ms
Avg time required for each read operation : 0.4 ms
Still, I got same readings.
I do not understand, why ZooKeeper gave such a poor write performance on some machines with windows 7. I tried these test cases with different jdk versions. But I am getting same result.
However, Read operation performance was same across all deployments.
Can someone please point out the possible cause for such extreme results ?
Thanks,
Amar
Re: Extremely different readings on different zookeeper deployments
Posted by ibrahim El-sanosi <ib...@gmail.com>.
This may help as well..
1. The write request requres consensus (three communcation steps where
read does not).
2. The more write requests the Zab has, the more messages
(Forwards/Proposals/ACKS/Commits messages) will be generated/sent between
Leader/Followers.
3. As (2) occurs, the replics (Leader/Followers) have more messages in
buffer (received queue), so the longer queue, the higer latency.
4. As write increases, the leader (potentially) becomes bottleneck.
N
On Tue, Feb 7, 2017 at 9:01 PM, Michael Han <ha...@cloudera.com> wrote:
> +1 on checking the disk set up first.
>
> Also, it is good to check the server logs on the Windows 7 boxes to see if
> there is anything obviously suspicious. In particular we log warnings if
> flushing the transaction to disk takes longer than a predefined threshold
> (1ms by default). Meanwhile another experiment could isolate the problem is
> to set zookeeper.forceSync
> <https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html> option to
> false on windows 7 boxes and see if it makes any difference.
>
> On Tue, Feb 7, 2017 at 11:20 AM, Camille Fournier <ca...@apache.org>
> wrote:
>
> > Disk writing speed is one of the major factors for zk write performance.
> Is
> > the disk setup the same across both of these machines? My guess is that
> is
> > a big factor.
> >
> > On Tue, Feb 7, 2017 at 2:24 AM, Amar Gajbhiye <Am...@1eq.com>
> > wrote:
> >
> > > Hi,
> > > I am working on a distributed system where I want to use consensus
> > > algorithm. I am evaluating ZooKeeper for the same.
> > > I am trying to quantify read and write operations for zookeeper for
> > > version 3.5.2.
> > >
> > > I am running following operations.
> > >
> > >
> > > 1. Create 50k Znodes with data
> > >
> > > 2. Read from those 50k znodes
> > >
> > >
> > > I tried this operations when zookeeper server was running on 4
> different
> > > remote servers in standalone mode. My client was always on same
> machine.
> > >
> > >
> > > 1. Test case 1
> > >
> > > Server configuration : Windows 8 server R2.
> > >
> > > jdk 1.7.0_80
> > >
> > > Avg time required for each write operations : 1.8 ms
> > >
> > > Avg time required for each read operations : 0.9 ms
> > >
> > >
> > >
> > >
> > > 2. Test case 2
> > >
> > > Server configuration : Windows 8 server R2.
> > >
> > > jdk 1.7.0_21
> > >
> > > Avg time required for each write operation : 1.7 ms
> > >
> > > Avg time required for each read operation : 0.9 ms
> > >
> > >
> > >
> > >
> > >
> > > 3. Test case 3
> > >
> > > Server configuration : Windows 7
> > >
> > > jdk 1.7.0_25
> > >
> > > Avg time required for each write operation : 35 ms
> > >
> > > Avg time required for each read operation : 0.8 ms
> > >
> > >
> > >
> > >
> > >
> > > 4. Test case 4
> > >
> > > Server configuration : Windows 7
> > >
> > > jdk1.8.0_101
> > >
> > > Avg time required for each write operation : 40 ms
> > >
> > > Avg time required for each read operation : 0.9 ms
> > >
> > >
> > >
> > >
> > >
> > > 5. To get network latency out of picture, I kept both client and
> > > server on same machine with following configuration
> > >
> > > Server configuration : Windows 7
> > >
> > > jdk1.8.0_101
> > >
> > > Avg time required for each write operation : 30 ms
> > >
> > > Avg time required for each read operation : 0.4 ms
> > >
> > > Still, I got same readings.
> > >
> > >
> > >
> > > I do not understand, why ZooKeeper gave such a poor write performance
> on
> > > some machines with windows 7. I tried these test cases with different
> jdk
> > > versions. But I am getting same result.
> > >
> > >
> > >
> > > However, Read operation performance was same across all deployments.
> > >
> > >
> > >
> > > Can someone please point out the possible cause for such extreme
> results
> > ?
> > >
> > >
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Amar
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
>
>
>
> --
> Cheers
> Michael.
>
Re: Extremely different readings on different zookeeper deployments
Posted by Michael Han <ha...@cloudera.com>.
+1 on checking the disk set up first.
Also, it is good to check the server logs on the Windows 7 boxes to see if
there is anything obviously suspicious. In particular we log warnings if
flushing the transaction to disk takes longer than a predefined threshold
(1ms by default). Meanwhile another experiment could isolate the problem is
to set zookeeper.forceSync
<https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html> option to
false on windows 7 boxes and see if it makes any difference.
On Tue, Feb 7, 2017 at 11:20 AM, Camille Fournier <ca...@apache.org>
wrote:
> Disk writing speed is one of the major factors for zk write performance. Is
> the disk setup the same across both of these machines? My guess is that is
> a big factor.
>
> On Tue, Feb 7, 2017 at 2:24 AM, Amar Gajbhiye <Am...@1eq.com>
> wrote:
>
> > Hi,
> > I am working on a distributed system where I want to use consensus
> > algorithm. I am evaluating ZooKeeper for the same.
> > I am trying to quantify read and write operations for zookeeper for
> > version 3.5.2.
> >
> > I am running following operations.
> >
> >
> > 1. Create 50k Znodes with data
> >
> > 2. Read from those 50k znodes
> >
> >
> > I tried this operations when zookeeper server was running on 4 different
> > remote servers in standalone mode. My client was always on same machine.
> >
> >
> > 1. Test case 1
> >
> > Server configuration : Windows 8 server R2.
> >
> > jdk 1.7.0_80
> >
> > Avg time required for each write operations : 1.8 ms
> >
> > Avg time required for each read operations : 0.9 ms
> >
> >
> >
> >
> > 2. Test case 2
> >
> > Server configuration : Windows 8 server R2.
> >
> > jdk 1.7.0_21
> >
> > Avg time required for each write operation : 1.7 ms
> >
> > Avg time required for each read operation : 0.9 ms
> >
> >
> >
> >
> >
> > 3. Test case 3
> >
> > Server configuration : Windows 7
> >
> > jdk 1.7.0_25
> >
> > Avg time required for each write operation : 35 ms
> >
> > Avg time required for each read operation : 0.8 ms
> >
> >
> >
> >
> >
> > 4. Test case 4
> >
> > Server configuration : Windows 7
> >
> > jdk1.8.0_101
> >
> > Avg time required for each write operation : 40 ms
> >
> > Avg time required for each read operation : 0.9 ms
> >
> >
> >
> >
> >
> > 5. To get network latency out of picture, I kept both client and
> > server on same machine with following configuration
> >
> > Server configuration : Windows 7
> >
> > jdk1.8.0_101
> >
> > Avg time required for each write operation : 30 ms
> >
> > Avg time required for each read operation : 0.4 ms
> >
> > Still, I got same readings.
> >
> >
> >
> > I do not understand, why ZooKeeper gave such a poor write performance on
> > some machines with windows 7. I tried these test cases with different jdk
> > versions. But I am getting same result.
> >
> >
> >
> > However, Read operation performance was same across all deployments.
> >
> >
> >
> > Can someone please point out the possible cause for such extreme results
> ?
> >
> >
> >
> >
> >
> > Thanks,
> >
> > Amar
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
--
Cheers
Michael.
Re: Extremely different readings on different zookeeper deployments
Posted by Jordan Zimmerman <jo...@jordanzimmerman.com>.
+1 on this. ZooKeeper is not a database. It's a distributed coordinator. Maybe if you describe your use-case this list can help better.
-Jordan
> On Feb 7, 2017, at 2:51 PM, MrSaurabh113 . <sa...@gmail.com> wrote:
>
> Zookeeper is simply not built for write heavy use cases.
>
> It's the best when you have like 20% writes and 80% reads.
>
> Thanks,
> Saurabh
>
> On Feb 8, 2017 12:50 AM, "Camille Fournier" <ca...@apache.org> wrote:
>
> Disk writing speed is one of the major factors for zk write performance. Is
> the disk setup the same across both of these machines? My guess is that is
> a big factor.
>
> On Tue, Feb 7, 2017 at 2:24 AM, Amar Gajbhiye <Am...@1eq.com> wrote:
>
>> Hi,
>> I am working on a distributed system where I want to use consensus
>> algorithm. I am evaluating ZooKeeper for the same.
>> I am trying to quantify read and write operations for zookeeper for
>> version 3.5.2.
>>
>> I am running following operations.
>>
>>
>> 1. Create 50k Znodes with data
>>
>> 2. Read from those 50k znodes
>>
>>
>> I tried this operations when zookeeper server was running on 4 different
>> remote servers in standalone mode. My client was always on same machine.
>>
>>
>> 1. Test case 1
>>
>> Server configuration : Windows 8 server R2.
>>
>> jdk 1.7.0_80
>>
>> Avg time required for each write operations : 1.8 ms
>>
>> Avg time required for each read operations : 0.9 ms
>>
>>
>>
>>
>> 2. Test case 2
>>
>> Server configuration : Windows 8 server R2.
>>
>> jdk 1.7.0_21
>>
>> Avg time required for each write operation : 1.7 ms
>>
>> Avg time required for each read operation : 0.9 ms
>>
>>
>>
>>
>>
>> 3. Test case 3
>>
>> Server configuration : Windows 7
>>
>> jdk 1.7.0_25
>>
>> Avg time required for each write operation : 35 ms
>>
>> Avg time required for each read operation : 0.8 ms
>>
>>
>>
>>
>>
>> 4. Test case 4
>>
>> Server configuration : Windows 7
>>
>> jdk1.8.0_101
>>
>> Avg time required for each write operation : 40 ms
>>
>> Avg time required for each read operation : 0.9 ms
>>
>>
>>
>>
>>
>> 5. To get network latency out of picture, I kept both client and
>> server on same machine with following configuration
>>
>> Server configuration : Windows 7
>>
>> jdk1.8.0_101
>>
>> Avg time required for each write operation : 30 ms
>>
>> Avg time required for each read operation : 0.4 ms
>>
>> Still, I got same readings.
>>
>>
>>
>> I do not understand, why ZooKeeper gave such a poor write performance on
>> some machines with windows 7. I tried these test cases with different jdk
>> versions. But I am getting same result.
>>
>>
>>
>> However, Read operation performance was same across all deployments.
>>
>>
>>
>> Can someone please point out the possible cause for such extreme results ?
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Amar
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
Re: Extremely different readings on different zookeeper deployments
Posted by "MrSaurabh113 ." <sa...@gmail.com>.
Zookeeper is simply not built for write heavy use cases.
It's the best when you have like 20% writes and 80% reads.
Thanks,
Saurabh
On Feb 8, 2017 12:50 AM, "Camille Fournier" <ca...@apache.org> wrote:
Disk writing speed is one of the major factors for zk write performance. Is
the disk setup the same across both of these machines? My guess is that is
a big factor.
On Tue, Feb 7, 2017 at 2:24 AM, Amar Gajbhiye <Am...@1eq.com> wrote:
> Hi,
> I am working on a distributed system where I want to use consensus
> algorithm. I am evaluating ZooKeeper for the same.
> I am trying to quantify read and write operations for zookeeper for
> version 3.5.2.
>
> I am running following operations.
>
>
> 1. Create 50k Znodes with data
>
> 2. Read from those 50k znodes
>
>
> I tried this operations when zookeeper server was running on 4 different
> remote servers in standalone mode. My client was always on same machine.
>
>
> 1. Test case 1
>
> Server configuration : Windows 8 server R2.
>
> jdk 1.7.0_80
>
> Avg time required for each write operations : 1.8 ms
>
> Avg time required for each read operations : 0.9 ms
>
>
>
>
> 2. Test case 2
>
> Server configuration : Windows 8 server R2.
>
> jdk 1.7.0_21
>
> Avg time required for each write operation : 1.7 ms
>
> Avg time required for each read operation : 0.9 ms
>
>
>
>
>
> 3. Test case 3
>
> Server configuration : Windows 7
>
> jdk 1.7.0_25
>
> Avg time required for each write operation : 35 ms
>
> Avg time required for each read operation : 0.8 ms
>
>
>
>
>
> 4. Test case 4
>
> Server configuration : Windows 7
>
> jdk1.8.0_101
>
> Avg time required for each write operation : 40 ms
>
> Avg time required for each read operation : 0.9 ms
>
>
>
>
>
> 5. To get network latency out of picture, I kept both client and
> server on same machine with following configuration
>
> Server configuration : Windows 7
>
> jdk1.8.0_101
>
> Avg time required for each write operation : 30 ms
>
> Avg time required for each read operation : 0.4 ms
>
> Still, I got same readings.
>
>
>
> I do not understand, why ZooKeeper gave such a poor write performance on
> some machines with windows 7. I tried these test cases with different jdk
> versions. But I am getting same result.
>
>
>
> However, Read operation performance was same across all deployments.
>
>
>
> Can someone please point out the possible cause for such extreme results ?
>
>
>
>
>
> Thanks,
>
> Amar
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
Re: Extremely different readings on different zookeeper deployments
Posted by Camille Fournier <ca...@apache.org>.
Disk writing speed is one of the major factors for zk write performance. Is
the disk setup the same across both of these machines? My guess is that is
a big factor.
On Tue, Feb 7, 2017 at 2:24 AM, Amar Gajbhiye <Am...@1eq.com> wrote:
> Hi,
> I am working on a distributed system where I want to use consensus
> algorithm. I am evaluating ZooKeeper for the same.
> I am trying to quantify read and write operations for zookeeper for
> version 3.5.2.
>
> I am running following operations.
>
>
> 1. Create 50k Znodes with data
>
> 2. Read from those 50k znodes
>
>
> I tried this operations when zookeeper server was running on 4 different
> remote servers in standalone mode. My client was always on same machine.
>
>
> 1. Test case 1
>
> Server configuration : Windows 8 server R2.
>
> jdk 1.7.0_80
>
> Avg time required for each write operations : 1.8 ms
>
> Avg time required for each read operations : 0.9 ms
>
>
>
>
> 2. Test case 2
>
> Server configuration : Windows 8 server R2.
>
> jdk 1.7.0_21
>
> Avg time required for each write operation : 1.7 ms
>
> Avg time required for each read operation : 0.9 ms
>
>
>
>
>
> 3. Test case 3
>
> Server configuration : Windows 7
>
> jdk 1.7.0_25
>
> Avg time required for each write operation : 35 ms
>
> Avg time required for each read operation : 0.8 ms
>
>
>
>
>
> 4. Test case 4
>
> Server configuration : Windows 7
>
> jdk1.8.0_101
>
> Avg time required for each write operation : 40 ms
>
> Avg time required for each read operation : 0.9 ms
>
>
>
>
>
> 5. To get network latency out of picture, I kept both client and
> server on same machine with following configuration
>
> Server configuration : Windows 7
>
> jdk1.8.0_101
>
> Avg time required for each write operation : 30 ms
>
> Avg time required for each read operation : 0.4 ms
>
> Still, I got same readings.
>
>
>
> I do not understand, why ZooKeeper gave such a poor write performance on
> some machines with windows 7. I tried these test cases with different jdk
> versions. But I am getting same result.
>
>
>
> However, Read operation performance was same across all deployments.
>
>
>
> Can someone please point out the possible cause for such extreme results ?
>
>
>
>
>
> Thanks,
>
> Amar
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>