You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hbase.apache.org by Ronen Itkin <ro...@taykey.com> on 2011/10/04 20:06:48 UTC

HBase write latency issues with hdfs replication of 3

Hi all!

I am getting really bad write performance when writing to HBase that relies
on hdfs (with replection parameter of 3) in Amazone Web Services
environment.
I am using Cloudera CDH3u1 distribution for all of the components.

Here are some benchmarks:

*Cluster 1:*
Back in my office (not on an Amazon instance):

   - 1 server (3GB RAM, 1 CPU unit) installed with a local Hadoop cluster
   (NameNode,DatNode,JTracker,DataNode), HBase cluster (HMaster,HRegion) and a
   local Zookeeper - all with default configuration.
   - HDFS replication parameter = *1*
   - Benchmark result: *writing 10,000 records in ~1 second*

*Cluster 2:*
On Amazon EC2 environment

   - 1 server - small instance (1.7GB RAM, 1 CPU unit) installed with a
   local Hadoop cluster (NameNode,DatNode,JTracker,DataNode), HBase cluster
   (HMaster,HRegion) and a local Zookeeper - all with default configuration.
   - HDFS replication parameter = *1*
   - Benchmark result: *writing 10,000 records in ~8 seconds*

*Cluster 3:*
On Amazon EC2 environment

   - 1 server - large instance (7.5GB RAM, 4 CPU Units) installed Hadoop
   NameNode, JotTracker, HBaseMaster, ZooKeeper.
   - 3 servers - xlarge instances (15GB RAM, 8 CPU Units  - each instance)
   installed Hadoop with DataNode, TaskTracker, HBaseRegionServer.
   - 1 server -  large instance (7.5GB RAM, 4 CPU Units) installed Hadoop
   SecondaryNameNode, HBaseBackupMaster, Zookeeper.
   - 1 server - small instance (1.7GB RAM, 1 CPU unit) installed with
   Zookeeper.
   - First run      - HDFS replication parameter = *1*  -->  Benchmark
   result: *writing 10,000 records in ~11 seconds*
   - Second run - HDFS replication parameter = *3*  -->  Benchmark
result: *writing
   10,000 records in ~58 seconds*


*Questions:*

   - Cluster 1 compared to cluster 2 - Why there is a huge difference in
   performance between Amazon environment and my back office environments ?? as
   far as I know the difference should be only the I/O, because Amazon's
   partitions are not really local, and yet the difference is huge!
   - Cluster 3 - Why does it make such a huge difference if the replication
   is set to tree? isnt it supposed to me transspar







-- 
*
Ronen Itkin*
Taykey | www.taykey.com

Re: HBase write latency issues with hdfs replication of 3

Posted by Jean-Daniel Cryans <jd...@apache.org>.

Inline at the end.

J-D

On Tue, Oct 4, 2011 at 11:06 AM, Ronen Itkin <ro...@taykey.com> wrote:
> Hi all!
>
> I am getting really bad write performance when writing to HBase that relies
> on hdfs (with replection parameter of 3) in Amazone Web Services
> environment.
> I am using Cloudera CDH3u1 distribution for all of the components.
>
> Here are some benchmarks:
>
> *Cluster 1:*
> Back in my office (not on an Amazon instance):
>
>   - 1 server (3GB RAM, 1 CPU unit) installed with a local Hadoop cluster
>   (NameNode,DatNode,JTracker,DataNode), HBase cluster (HMaster,HRegion) and a
>   local Zookeeper - all with default configuration.
>   - HDFS replication parameter = *1*
>   - Benchmark result: *writing 10,000 records in ~1 second*
>
> *Cluster 2:*
> On Amazon EC2 environment
>
>   - 1 server - small instance (1.7GB RAM, 1 CPU unit) installed with a
>   local Hadoop cluster (NameNode,DatNode,JTracker,DataNode), HBase cluster
>   (HMaster,HRegion) and a local Zookeeper - all with default configuration.
>   - HDFS replication parameter = *1*
>   - Benchmark result: *writing 10,000 records in ~8 seconds*
>
> *Cluster 3:*
> On Amazon EC2 environment
>
>   - 1 server - large instance (7.5GB RAM, 4 CPU Units) installed Hadoop
>   NameNode, JotTracker, HBaseMaster, ZooKeeper.
>   - 3 servers - xlarge instances (15GB RAM, 8 CPU Units  - each instance)
>   installed Hadoop with DataNode, TaskTracker, HBaseRegionServer.
>   - 1 server -  large instance (7.5GB RAM, 4 CPU Units) installed Hadoop
>   SecondaryNameNode, HBaseBackupMaster, Zookeeper.
>   - 1 server - small instance (1.7GB RAM, 1 CPU unit) installed with
>   Zookeeper.
>   - First run      - HDFS replication parameter = *1*  -->  Benchmark
>   result: *writing 10,000 records in ~11 seconds*
>   - Second run - HDFS replication parameter = *3*  -->  Benchmark
> result: *writing
>   10,000 records in ~58 seconds*
>
>
> *Questions:*
>
>   - Cluster 1 compared to cluster 2 - Why there is a huge difference in
>   performance between Amazon environment and my back office environments ?? as
>   far as I know the difference should be only the I/O, because Amazon's
>   partitions are not really local, and yet the difference is huge!

You need to read more on EC2. Basically the smaller the instance the
smaller the IO share it has. It is *not* like having a full machine
just for you (until you get to the bigger instances, but even then
it's still a shared environment).

>   - Cluster 3 - Why does it make such a huge difference if the replication
>   is set to tree? Isnt it supposed to be transparent ? and get the
>    same performance? because the client's application interacts with the name
>    node and waits to receive only one ACK per packet? Does the NameNode sends
>    that ACK after the first successful write or after the whole 3 replications
>    are ready? is that configurable? maybe its the client's application fault?

Ok so first, replicating to 3 nodes takes longer than writing to just
one. Replication in HDFS works in a pipeline, client writes to
datanode1, which writes to datanode2, which writes to datanode3 and
then the ACK comes all the way back.

Second, the IO performance you're getting in EC2 is probably less than
what you think. Once you have 3 machines involved which need to talk
to each other, you might be taking a serious hit. Also I would
recommend running each tests a few times because another about EC2 is
that performance varies... sometimes a lot!

>
>
>
>
>
>
>
> --
> *
> Ronen Itkin*
> Taykey | www.taykey.com
>

Re: HBase write latency issues with hdfs replication of 3

Posted by Ronen Itkin <ro...@taykey.com>.

Sorry, I sent it by mistake before I have finished writing.
here is the full mail.

*Cluster 1:*
> Back in my office (not on an Amazon instance):
>
>    - 1 server (3GB RAM, 1 CPU unit) installed with a local Hadoop cluster
>    (NameNode,DatNode,JTracker,DataNode), HBase cluster (HMaster,HRegion) and a
>    local Zookeeper - all with default configuration.
>    - HDFS replication parameter = *1*
>    - Benchmark result: *writing 10,000 records in ~1 second*
>
> *Cluster 2:*
> On Amazon EC2 environment
>
>    - 1 server - small instance (1.7GB RAM, 1 CPU unit) installed with a
>    local Hadoop cluster (NameNode,DatNode,JTracker,DataNode), HBase cluster
>    (HMaster,HRegion) and a local Zookeeper - all with default configuration.
>    - HDFS replication parameter = *1*
>    - Benchmark result: *writing 10,000 records in ~8 seconds*
>
> *Cluster 3:*
> On Amazon EC2 environment
>
>    - 1 server - large instance (7.5GB RAM, 4 CPU Units) installed Hadoop
>    NameNode, JotTracker, HBaseMaster, ZooKeeper.
>    - 3 servers - xlarge instances (15GB RAM, 8 CPU Units  - each instance)
>    installed Hadoop with DataNode, TaskTracker, HBaseRegionServer.
>    - 1 server -  large instance (7.5GB RAM, 4 CPU Units) installed Hadoop
>    SecondaryNameNode, HBaseBackupMaster, Zookeeper.
>    - 1 server - small instance (1.7GB RAM, 1 CPU unit) installed with
>    Zookeeper.
>    - First run      - HDFS replication parameter = *1*  -->  Benchmark
>    result: *writing 10,000 records in ~11 seconds*
>    - Second run - HDFS replication parameter = *3*  -->  Benchmark result:
>    *writing 10,000 records in ~58 seconds*
>
>
> *Questions:*
>
>    - Cluster 1 compared to cluster 2 - Why there is a huge difference in
>    performance between Amazon environment and my back office environments ?? as
>    far as I know the difference should be only the I/O, because Amazon's
>    partitions are not really local, and yet the difference is huge!
>    - Cluster 3 - Why does it make such a huge difference if the
>    replication is set to tree? Isnt it supposed to be transparent ? and get the
>    same performance? because the client's application interacts with the name
>    node and waits to receive only one ACK per packet? Does the NameNode sends
>    that ACK after the first successful write or after the whole 3 replications
>    are ready? is that configurable? maybe its the client's application fault?
>
>     Sorry for the long e-mail!

> Thanks alot!!
>
*
> Ronen Itkin*
> Taykey | www.taykey.com
>
>