Some basic questions on replication

[Hari Sreekumar <hs...@clickable.com>]

Hi,

 I have some pretty basic stuff on replication that I am no very clear
about, even after reading the online docs..

1. My understanding is that replication factor of x means any block of data
in HDFS will be available, given enough time, at x different nodes. I have a
confusion whether it is x nodes or x locations or x disks? e.g, if I have
replication set to 3 on a single node setup with one physical disk, we'll
have the same data at 3 locations on the hard drive? What if I have 3 disks
on the node?

2. If I have a 3 node setup with replication set to 1, and I upload into it
a 3 GB file, it means that 1 GB of the file, approximately, will be
available on each node, right?

3. If I run a mapreduce job on the 3 GB file above, will there be any data
transfer between the nodes for the map phase? The optimizer will try to
assign tasks in a way that each node uses the locally available data, so
each Node will run the map function based on locally available data, right?
In the reduce phase, of course, the map outputs will be shuffled.

I am asking these questions because in a recent test Map-reduce job I ran on
a 2.13 GB file (3 node cluster), the job competed in 40 s with
replication=3, but took 1 min 45 s with replication=1. What could be the
reason for this? Can network latency be a reason? The job is simply an
aggregation where map returns IntWritable(1)s and reduce just sums it up.

Thanks in advance,
Hari

Re: Some basic questions on replication

[Harsh J <qw...@gmail.com>]

Hello again,

On Fri, Nov 5, 2010 at 12:52 AM, Hari Sreekumar
<hs...@clickable.com> wrote:
> Hi Harsh,
>
>             Thanks for the reply. So if I have a 2048 MB file with 64 MB
> block size (32 blocks) with replication 3, then I'll have 96 blocks of the
> file on HDFS, with no two similar blocks being on the same datanode. Also,
> if I change the dfs.replication property, does it effect files already in
> HDFS or is it valid only for new files that will be uploaded into HDFS? Is
> there a way to rebalance the cluster based on the new replication factor?

Replication is file-based. It will not affect existing files, if you
restart DataNodes with a different replication factor value.

There's a "setrep" command you could use to reset replication value of
files. See: http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#setrep

>
> And if I have replication set to 3, do all the 3 disk writes happen
> simultaneously or is there some background process which does the
> replication? If not, then increasing replication would lead to more writes
> and thus reduce performance of any write-intensive job, am I right?

They do not happen simultaneously. The NameNode determines the work of
replicating blocks to DataNodes every nth interval, set by
"dfs.namenode.replication.interval" to 3s by default. I believe that
the load of the nodes are also considered while attempting to assign a
replication work between DataNodes.

I haven't seen increasing/decreasing replication factors affecting the
performance of MapReduce jobs when it comes to writing, but yes I
suppose it could, in case of decommissioning a node and/or
re-balancing, lower the network transfer rates that the jobs may
utilize.


-- 
Harsh J
www.harshj.com

Re: Some basic questions on replication

[Hari Sreekumar <hs...@clickable.com>]

Hi Harsh,

             Thanks for the reply. So if I have a 2048 MB file with 64 MB
block size (32 blocks) with replication 3, then I'll have 96 blocks of the
file on HDFS, with no two similar blocks being on the same datanode. Also,
if I change the dfs.replication property, does it effect files already in
HDFS or is it valid only for new files that will be uploaded into HDFS? Is
there a way to rebalance the cluster based on the new replication factor?

And if I have replication set to 3, do all the 3 disk writes happen
simultaneously or is there some background process which does the
replication? If not, then increasing replication would lead to more writes
and thus reduce performance of any write-intensive job, am I right?

Thanks,
Hari

On Thu, Nov 4, 2010 at 11:59 PM, Harsh J <qw...@gmail.com> wrote:

> Hi,
>
> The following inline reply is from what I know so far.
>
> On Thu, Nov 4, 2010 at 11:24 PM, Hari Sreekumar
> <hs...@clickable.com> wrote:
> > Hi,
> >
> >  I have some pretty basic stuff on replication that I am no very clear
> > about, even after reading the online docs..
> >
> > 1. My understanding is that replication factor of x means any block of
> data
> > in HDFS will be available, given enough time, at x different nodes. I
> have a
> > confusion whether it is x nodes or x locations or x disks? e.g, if I have
> > replication set to 3 on a single node setup with one physical disk, we'll
> > have the same data at 3 locations on the hard drive? What if I have 3
> disks
> > on the node?
>
> A "factor" of 3 means that a block of HDFS data (whose replication
> factor is also set to 3) will be available at 3 nodes. That means, if
> a file is worth 3 blocks, there will be 3x3=9 blocks available in your
> cluster totally for that one file.
>
> Replication is between DataNodes. DataNodes can use a single disk or
> multiple disks, but the concept of replication applies to Node-level,
> not disk-level.
>
> If am right, on a single node, if you have replication as 3, it will
> still create one block cause only one DataNode is available. Perhaps
> it'll create the replicas once new DataNodes are recognized by the
> cluster.
>
> >
> > 2. If I have a 3 node setup with replication set to 1, and I upload into
> it
> > a 3 GB file, it means that 1 GB of the file, approximately, will be
> > available on each node, right?
>
> If your block size is set to 1 GB, then yes.
>
> But know that if you are uploading into the HDFS from a node that has
> a DataNode service on it, it will contain all 3 blocks (an
> optimization). The best way to upload a file to HDFS would be to do so
> from a node that does not run a DataNode, as this will ensure a random
> distribution of blocks. (Correct me if am wrong).
>
> >
> > 3. If I run a mapreduce job on the 3 GB file above, will there be any
> data
> > transfer between the nodes for the map phase? The optimizer will try to
> > assign tasks in a way that each node uses the locally available data, so
> > each Node will run the map function based on locally available data,
> right?
> > In the reduce phase, of course, the map outputs will be shuffled.
>
> Data transfer will occur only for non-local and rack-local map tasks
> (which are usually run when TaskTracker slots are unavailable for maps
> to run locally on them).
>
> >
> > I am asking these questions because in a recent test Map-reduce job I ran
> on
> > a 2.13 GB file (3 node cluster), the job competed in 40 s with
> > replication=3, but took 1 min 45 s with replication=1. What could be the
> > reason for this? Can network latency be a reason? The job is simply an
> > aggregation where map returns IntWritable(1)s and reduce just sums it up.
>
> The time delay here could be related to several things. One could be
> that with replication=1, only one DataNode had your blocks (since you
> uploaded from a node running the service, as pointed above).
>
> But the best way to see why would be to see how many of your map tasks
> on the whole, were local. You can see this on the JobTracker Web UI as
> a counter (or on the CLI, after the job's succeeded).
>
> >
> > Thanks in advance,
> > Hari
> >
>
>
>
> --
> Harsh J
> www.harshj.com
>



-- 
Hari Shankar Sreekumar
Software Development Engineer
Clickable Inc.

Ph: +91-9968307501
Skype: hari.clickable

Re: Some basic questions on replication

[Harsh J <qw...@gmail.com>]

Hi,

The following inline reply is from what I know so far.

On Thu, Nov 4, 2010 at 11:24 PM, Hari Sreekumar
<hs...@clickable.com> wrote:
> Hi,
>
>  I have some pretty basic stuff on replication that I am no very clear
> about, even after reading the online docs..
>
> 1. My understanding is that replication factor of x means any block of data
> in HDFS will be available, given enough time, at x different nodes. I have a
> confusion whether it is x nodes or x locations or x disks? e.g, if I have
> replication set to 3 on a single node setup with one physical disk, we'll
> have the same data at 3 locations on the hard drive? What if I have 3 disks
> on the node?

A "factor" of 3 means that a block of HDFS data (whose replication
factor is also set to 3) will be available at 3 nodes. That means, if
a file is worth 3 blocks, there will be 3x3=9 blocks available in your
cluster totally for that one file.

Replication is between DataNodes. DataNodes can use a single disk or
multiple disks, but the concept of replication applies to Node-level,
not disk-level.

If am right, on a single node, if you have replication as 3, it will
still create one block cause only one DataNode is available. Perhaps
it'll create the replicas once new DataNodes are recognized by the
cluster.

>
> 2. If I have a 3 node setup with replication set to 1, and I upload into it
> a 3 GB file, it means that 1 GB of the file, approximately, will be
> available on each node, right?

If your block size is set to 1 GB, then yes.

But know that if you are uploading into the HDFS from a node that has
a DataNode service on it, it will contain all 3 blocks (an
optimization). The best way to upload a file to HDFS would be to do so
from a node that does not run a DataNode, as this will ensure a random
distribution of blocks. (Correct me if am wrong).

>
> 3. If I run a mapreduce job on the 3 GB file above, will there be any data
> transfer between the nodes for the map phase? The optimizer will try to
> assign tasks in a way that each node uses the locally available data, so
> each Node will run the map function based on locally available data, right?
> In the reduce phase, of course, the map outputs will be shuffled.

Data transfer will occur only for non-local and rack-local map tasks
(which are usually run when TaskTracker slots are unavailable for maps
to run locally on them).

>
> I am asking these questions because in a recent test Map-reduce job I ran on
> a 2.13 GB file (3 node cluster), the job competed in 40 s with
> replication=3, but took 1 min 45 s with replication=1. What could be the
> reason for this? Can network latency be a reason? The job is simply an
> aggregation where map returns IntWritable(1)s and reduce just sums it up.

The time delay here could be related to several things. One could be
that with replication=1, only one DataNode had your blocks (since you
uploaded from a node running the service, as pointed above).

But the best way to see why would be to see how many of your map tasks
on the whole, were local. You can see this on the JobTracker Web UI as
a counter (or on the CLI, after the job's succeeded).

>
> Thanks in advance,
> Hari
>



-- 
Harsh J
www.harshj.com