You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ChaoChun Liang <cc...@gmail.com> on 2007/09/05 10:58:15 UTC
Replication problem of HDFS
According the reference of
HDFS(http://lucene.apache.org/hadoop/hdfs_design.html),
a file in the HDFS will be split into one or more blocks and these blocks
are stored in
a set of Datanodes.
I put(set replication=1) a 2GB data set to a 5-nodes cluster, but found only
the
namenode increase the block numbers, others nodes keep the same value. It
means
all blocks copied to the namenode, none to datanodes. Is it correct?
ChaoChun
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12494269
Sent from the Hadoop Users mailing list archive at Nabble.com.
RE: Replication problem of HDFS
Posted by ChaoChun Liang <cc...@gmail.com>.
Yes, you are right. the namenode and datanode are in the same machine
and upload data into HDFS in the same one in my environment. I suppose
the HDFS will distribute these blocks to all others datanode(according the
HDFS reference), but it is not actually.
>>Inthis case, the only replica of the file will reside on the Datanode that
is
>>local to the client.
So, does it conflict with the HDFS reference? (a file in the HDFS will be
split into
one or more blocks and these blocks are stored in a set of Datanodes. )
What kind of uploading to let all data/files store into the datanodes(not a
single one)?
ChaoChun
Dhruba Borthakur wrote:
>
> Hi ChaoChun,
>
> I do not fully understand your problem. I am guessing that you are running
> a
> Datanode on the same machine as the Namenode. I am also guessing that you
> are using the Namenode machine as a client to upload a file into HDFS. In
> this case, the only replica of the file will reside on the Datanode that
> is
> local to the client.
>
> Thanks,
> dhruba
>
> -----Original Message-----
> From: ChaoChun Liang [mailto:ccliangnn@gmail.com]
> Sent: Wednesday, September 05, 2007 1:58 AM
> To: hadoop-user@lucene.apache.org
> Subject: Replication problem of HDFS
>
>
> According the reference of
> HDFS(http://lucene.apache.org/hadoop/hdfs_design.html),
> a file in the HDFS will be split into one or more blocks and these blocks
> are stored in
> a set of Datanodes.
>
> I put(set replication=1) a 2GB data set to a 5-nodes cluster, but found
> only
> the
> namenode increase the block numbers, others nodes keep the same value. It
> means
> all blocks copied to the namenode, none to datanodes. Is it correct?
>
> ChaoChun
> --
> View this message in context:
> http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12494269
> Sent from the Hadoop Users mailing list archive at Nabble.com.
>
>
>
>
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12514090
Sent from the Hadoop Users mailing list archive at Nabble.com.
RE: Replication problem of HDFS
Posted by Dhruba Borthakur <dh...@yahoo-inc.com>.
Hi ChaoChun,
I do not fully understand your problem. I am guessing that you are running a
Datanode on the same machine as the Namenode. I am also guessing that you
are using the Namenode machine as a client to upload a file into HDFS. In
this case, the only replica of the file will reside on the Datanode that is
local to the client.
Thanks,
dhruba
-----Original Message-----
From: ChaoChun Liang [mailto:ccliangnn@gmail.com]
Sent: Wednesday, September 05, 2007 1:58 AM
To: hadoop-user@lucene.apache.org
Subject: Replication problem of HDFS
According the reference of
HDFS(http://lucene.apache.org/hadoop/hdfs_design.html),
a file in the HDFS will be split into one or more blocks and these blocks
are stored in
a set of Datanodes.
I put(set replication=1) a 2GB data set to a 5-nodes cluster, but found only
the
namenode increase the block numbers, others nodes keep the same value. It
means
all blocks copied to the namenode, none to datanodes. Is it correct?
ChaoChun
--
View this message in context:
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12494269
Sent from the Hadoop Users mailing list archive at Nabble.com.
RE: Replication problem of HDFS
Posted by Dhruba Borthakur <dh...@yahoo-inc.com>.
This is expected behaviour. Since you have 4 datanodes, it might make sense
to bump up the replication factor to 2 or higher. Then you would see other
Datanodes getting filled up with blocks.
Thanks,
dhruba
-----Original Message-----
From: ChaoChun Liang [mailto:ccliangnn@gmail.com]
Sent: Wednesday, September 12, 2007 11:12 PM
To: hadoop-user@lucene.apache.org
Subject: Re: Replication problem of HDFS
Thanks for your detail example and explanation.
The problem what I met is, all split blocks stored in the same datanode,
that is, (A1, A2, A3) stored in the same datanode in your example.
My test case is putting (by "hadoop fs -put" command) a file about 1GB to
HDFS
with 4 datanodes, where the namenode and datanode are in the same machine).
dfs.block.size=67108864 and dfs.replication=1 in the hadoop-site.xml
And uploading from the namenode(i.e A3 as following) machine.
P.S. the namenode and datanode in the same machine, A3.
The datanode status "before" the uploading process is
------------------------------------------------------------------------
Node Last Contact Admin State Size (GB) Used (%)
Blocks
A1 2 In Service 37.23 18.94 1
A2 2 In Service 36.06 19.30 1
A3 1 In Service 39.06 70.13 18
A4 1 In Service 39.06 18.52 1
The datanode status "after" the uploading process is
------------------------------------------------------------------------
A1 2 In Service 37.23 18.94 1
A2 2 In Service 36.06 19.30 1
A3 1 In Service 39.06 71.95 35
A4 1 In Service 39.06 18.52 1
You can see that blocks increases only in the A3 node (by 17 blocks, from 18
to 35),
and the block numbers is others datanodes are the same and not changed.
Is it look something wrong? Or it is the configuration problem.
ChaoChun
Ted Dunning-3 wrote:
>
>
> Your question is very hard to understand. The problem may be the names of
> the different kinds of server.
>
> There is one namenode and there are many datanodes.
>
> Each file is divided into one or more blocks. By default the block has a
> maximum size of 64MB. Each block from a file is stored on one or more
> datanodes. The number of datanodes holding each block is called
> replication
> factor. The namenode holds information about what blocks are in each
> file.
> The namenode also contains information about what blocks each datanode
> holds.
>
> As an example, consider that you have 3 files called A, B, and C. Each
> file
> is 150MB so they have two full size blocks (A1, A2, B1, B2, C1, C2) and
> one
> partial block that is 22MB in size (A3, B3, C3).
>
> Suppose that replication factor is 1 for A, 2 for B and 3 for C.
>
> One possible state of five datanodes is this:
>
> Datanode1:
> A1, B2, C3, C1
>
> Datanode2:
> A2, C2, B2
>
> Datanode3:
> A3, C1, C3, B1
>
> Datanode4:
> B1, C1, C2, B3
>
> Datanode5:
> B3, C2, C3
>
> The namenode would contain this information:
>
> A -> (A1, A2, A3)
> B -> (B1, B2, B3)
> C -> (C1, C2, C3)
>
> A1 -> (Datanode1)
> B1 -> (Datanode3, Datanode4)
> C1 -> (Datanode1, Datanode3, Datanode4)
> ... And so on ...
>
> Does that help?
>
> On 9/10/07 8:04 PM, "ChaoChun Liang" <cc...@gmail.com> wrote:
>
>
>>
>> In my application, whether M blocks(described as above) exist in the name
>> datanode(i.e. each database
>> owns a completed M block), or shared M blocks for datanodes in the HDFS
>> is
>> important for us.
>>
>> If these M blocks could be shared, we may use the HDFS, otherswise we may
>> condiser the local file system
>> for the map/reduce processing.
>
>
>
--
View this message in context:
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12649233
Sent from the Hadoop Users mailing list archive at Nabble.com.
Re: Replication problem of HDFS
Posted by Ted Dunning <td...@veoh.com>.
This is a knonw behavior (a feature, even). When yu write on a datanode, it
prefers to put the data on that node because it is local.
To avoid this r
un the put on a non-datanode.
Or do the put with a higher replication and drop the replication after the
put.
Or use distcp if all of the data nodes have access to the same data (perhaps
via nfs).
On 9/12/07 11:11 PM, "ChaoChun Liang" <cc...@gmail.com> wrote:
>
> Thanks for your detail example and explanation.
>
> The problem what I met is, all split blocks stored in the same datanode,
> that is, (A1, A2, A3) stored in the same datanode in your example.
>
> My test case is putting (by "hadoop fs -put" command) a file about 1GB to
> HDFS
...
>
> Is it look something wrong? Or it is the configuration problem.
Re: Replication problem of HDFS
Posted by ChaoChun Liang <cc...@gmail.com>.
Thanks for your detail example and explanation.
The problem what I met is, all split blocks stored in the same datanode,
that is, (A1, A2, A3) stored in the same datanode in your example.
My test case is putting (by "hadoop fs -put" command) a file about 1GB to
HDFS
with 4 datanodes, where the namenode and datanode are in the same machine).
dfs.block.size=67108864 and dfs.replication=1 in the hadoop-site.xml
And uploading from the namenode(i.e A3 as following) machine.
P.S. the namenode and datanode in the same machine, A3.
The datanode status "before" the uploading process is
------------------------------------------------------------------------
Node Last Contact Admin State Size (GB) Used (%) Blocks
A1 2 In Service 37.23 18.94 1
A2 2 In Service 36.06 19.30 1
A3 1 In Service 39.06 70.13 18
A4 1 In Service 39.06 18.52 1
The datanode status "after" the uploading process is
------------------------------------------------------------------------
A1 2 In Service 37.23 18.94 1
A2 2 In Service 36.06 19.30 1
A3 1 In Service 39.06 71.95 35
A4 1 In Service 39.06 18.52 1
You can see that blocks increases only in the A3 node (by 17 blocks, from 18
to 35),
and the block numbers is others datanodes are the same and not changed.
Is it look something wrong? Or it is the configuration problem.
ChaoChun
Ted Dunning-3 wrote:
>
>
> Your question is very hard to understand. The problem may be the names of
> the different kinds of server.
>
> There is one namenode and there are many datanodes.
>
> Each file is divided into one or more blocks. By default the block has a
> maximum size of 64MB. Each block from a file is stored on one or more
> datanodes. The number of datanodes holding each block is called
> replication
> factor. The namenode holds information about what blocks are in each
> file.
> The namenode also contains information about what blocks each datanode
> holds.
>
> As an example, consider that you have 3 files called A, B, and C. Each
> file
> is 150MB so they have two full size blocks (A1, A2, B1, B2, C1, C2) and
> one
> partial block that is 22MB in size (A3, B3, C3).
>
> Suppose that replication factor is 1 for A, 2 for B and 3 for C.
>
> One possible state of five datanodes is this:
>
> Datanode1:
> A1, B2, C3, C1
>
> Datanode2:
> A2, C2, B2
>
> Datanode3:
> A3, C1, C3, B1
>
> Datanode4:
> B1, C1, C2, B3
>
> Datanode5:
> B3, C2, C3
>
> The namenode would contain this information:
>
> A -> (A1, A2, A3)
> B -> (B1, B2, B3)
> C -> (C1, C2, C3)
>
> A1 -> (Datanode1)
> B1 -> (Datanode3, Datanode4)
> C1 -> (Datanode1, Datanode3, Datanode4)
> ... And so on ...
>
> Does that help?
>
> On 9/10/07 8:04 PM, "ChaoChun Liang" <cc...@gmail.com> wrote:
>
>
>>
>> In my application, whether M blocks(described as above) exist in the name
>> datanode(i.e. each database
>> owns a completed M block), or shared M blocks for datanodes in the HDFS
>> is
>> important for us.
>>
>> If these M blocks could be shared, we may use the HDFS, otherswise we may
>> condiser the local file system
>> for the map/reduce processing.
>
>
>
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12649233
Sent from the Hadoop Users mailing list archive at Nabble.com.
Re: Replication problem of HDFS
Posted by Ted Dunning <td...@veoh.com>.
Your question is very hard to understand. The problem may be the names of
the different kinds of server.
There is one namenode and there are many datanodes.
Each file is divided into one or more blocks. By default the block has a
maximum size of 64MB. Each block from a file is stored on one or more
datanodes. The number of datanodes holding each block is called replication
factor. The namenode holds information about what blocks are in each file.
The namenode also contains information about what blocks each datanode
holds.
As an example, consider that you have 3 files called A, B, and C. Each file
is 150MB so they have two full size blocks (A1, A2, B1, B2, C1, C2) and one
partial block that is 22MB in size (A3, B3, C3).
Suppose that replication factor is 1 for A, 2 for B and 3 for C.
One possible state of five datanodes is this:
Datanode1:
A1, B2, C3, C1
Datanode2:
A2, C2, B2
Datanode3:
A3, C1, C3, B1
Datanode4:
B1, C1, C2, B3
Datanode5:
B3, C2, C3
The namenode would contain this information:
A -> (A1, A2, A3)
B -> (B1, B2, B3)
C -> (C1, C2, C3)
A1 -> (Datanode1)
B1 -> (Datanode3, Datanode4)
C1 -> (Datanode1, Datanode3, Datanode4)
... And so on ...
Does that help?
On 9/10/07 8:04 PM, "ChaoChun Liang" <cc...@gmail.com> wrote:
>
> In my application, whether M blocks(described as above) exist in the name
> datanode(i.e. each database
> owns a completed M block), or shared M blocks for datanodes in the HDFS is
> important for us.
>
> If these M blocks could be shared, we may use the HDFS, otherswise we may
> condiser the local file system
> for the map/reduce processing.
RE: Replication problem of HDFS
Posted by ChaoChun Liang <cc...@gmail.com>.
Hello, Stu:
>> If "replication == 2" then it will make sure that 2 copies of each of the
>> M blocks exist on datanodes.
>>No, it means to replicate the file to N datanodes. The client is only used
to transfer files to/from >>Hadoop: it doesn't do any long term storage.
Sorry, the client what I means is the datanodes.
In my application, whether M blocks(described as above) exist in the name
datanode(i.e. each database
owns a completed M block), or shared M blocks for datanodes in the HDFS is
important for us.
If these M blocks could be shared, we may use the HDFS, otherswise we may
condiser the local file system
for the map/reduce processing.
ChaoChun
-----Original Message-----
From: ChaoChun Liang
Sent: Thursday, September 6, 2007 10:23pm
To: hadoop-user@lucene.apache.org
Subject: RE: Replication problem of HDFS
So, the upload process(from local file system to HDFS) will store all
blocks(split from the dataset,
said M split blocks) into a single node(depend on which client you put), not
to all datanodes.
And the "replication" means to replicate to N clients(if replication=N) and
each client owns
a completed/all M blocks. If I am wrong, please correct it. Thanks.
ChaoChun
Stu Hood-2 wrote:
>
> ChaoChun,
>
> Since you set the 'replication = 1' for the file, only 1 copy of the
> file's blocks will be stored in Hadoop. If you want all 5 machines to have
> copies of each block, then you would set 'replication = 5' for the file.
>
> The default for replication is 3.
>
> Thanks,
> Stu
>
>
--
View this message in context:
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12534839
Sent from the Hadoop Users mailing list archive at Nabble.com.
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12607008
Sent from the Hadoop Users mailing list archive at Nabble.com.
RE: Replication problem of HDFS
Posted by ChaoChun Liang <cc...@gmail.com>.
According the HDFS reference, it sounds right.
ChaoChun
Earney, Billy C. wrote:
>
> ChoaChun,
>
> I'm new to hadoop, but my understanding is that the data is divided into
> blocks, and that not all blocks need be on the same node. So if a file
> has 2 blocks the first block of a file could be on node 1 and the second
> block could be on node 2. From the link below, it seems that for each
> block, the client will contact the namenode and request one or more
> datanodes to store the block on.
>
> http://lucene.apache.org/hadoop/hdfs_design.html#Replication+Pipelining
>
> Is my understanding of the documentation correct?
>
>
> -----Original Message-----
> From: ChaoChun Liang [mailto:ccliangnn@gmail.com]
> Sent: Thursday, September 06, 2007 9:23 PM
> To: hadoop-user@lucene.apache.org
> Subject: RE: Replication problem of HDFS
>
>
> So, the upload process(from local file system to HDFS) will store all
> blocks(split from the dataset,
> said M split blocks) into a single node(depend on which client you put),
> not
> to all datanodes.
> And the "replication" means to replicate to N clients(if replication=N)
> and
> each client owns
> a completed/all M blocks. If I am wrong, please correct it. Thanks.
>
> ChaoChun
>
>
> Stu Hood-2 wrote:
>>
>> ChaoChun,
>>
>> Since you set the 'replication = 1' for the file, only 1 copy of the
>> file's blocks will be stored in Hadoop. If you want all 5 machines to
> have
>> copies of each block, then you would set 'replication = 5' for the
> file.
>>
>> The default for replication is 3.
>>
>> Thanks,
>> Stu
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a125348
> 39
> Sent from the Hadoop Users mailing list archive at Nabble.com.
>
>
>
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12607024
Sent from the Hadoop Users mailing list archive at Nabble.com.
RE: Replication problem of HDFS
Posted by Dhruba Borthakur <dh...@yahoo-inc.com>.
Hi ChoaChun,
Your explanation sounds right.
Thanks,
dhruba
-----Original Message-----
From: Earney, Billy C. [mailto:earney@umsystem.edu]
Sent: Monday, September 10, 2007 10:44 AM
To: hadoop-user@lucene.apache.org
Subject: RE: Replication problem of HDFS
ChoaChun,
I'm new to hadoop, but my understanding is that the data is divided into
blocks, and that not all blocks need be on the same node. So if a file
has 2 blocks the first block of a file could be on node 1 and the second
block could be on node 2. From the link below, it seems that for each
block, the client will contact the namenode and request one or more
datanodes to store the block on.
http://lucene.apache.org/hadoop/hdfs_design.html#Replication+Pipelining
Is my understanding of the documentation correct?
-----Original Message-----
From: ChaoChun Liang [mailto:ccliangnn@gmail.com]
Sent: Thursday, September 06, 2007 9:23 PM
To: hadoop-user@lucene.apache.org
Subject: RE: Replication problem of HDFS
So, the upload process(from local file system to HDFS) will store all
blocks(split from the dataset,
said M split blocks) into a single node(depend on which client you put),
not
to all datanodes.
And the "replication" means to replicate to N clients(if replication=N)
and
each client owns
a completed/all M blocks. If I am wrong, please correct it. Thanks.
ChaoChun
Stu Hood-2 wrote:
>
> ChaoChun,
>
> Since you set the 'replication = 1' for the file, only 1 copy of the
> file's blocks will be stored in Hadoop. If you want all 5 machines to
have
> copies of each block, then you would set 'replication = 5' for the
file.
>
> The default for replication is 3.
>
> Thanks,
> Stu
>
>
--
View this message in context:
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a125348
39
Sent from the Hadoop Users mailing list archive at Nabble.com.
RE: Replication problem of HDFS
Posted by "Earney, Billy C." <ea...@umsystem.edu>.
ChoaChun,
I'm new to hadoop, but my understanding is that the data is divided into
blocks, and that not all blocks need be on the same node. So if a file
has 2 blocks the first block of a file could be on node 1 and the second
block could be on node 2. From the link below, it seems that for each
block, the client will contact the namenode and request one or more
datanodes to store the block on.
http://lucene.apache.org/hadoop/hdfs_design.html#Replication+Pipelining
Is my understanding of the documentation correct?
-----Original Message-----
From: ChaoChun Liang [mailto:ccliangnn@gmail.com]
Sent: Thursday, September 06, 2007 9:23 PM
To: hadoop-user@lucene.apache.org
Subject: RE: Replication problem of HDFS
So, the upload process(from local file system to HDFS) will store all
blocks(split from the dataset,
said M split blocks) into a single node(depend on which client you put),
not
to all datanodes.
And the "replication" means to replicate to N clients(if replication=N)
and
each client owns
a completed/all M blocks. If I am wrong, please correct it. Thanks.
ChaoChun
Stu Hood-2 wrote:
>
> ChaoChun,
>
> Since you set the 'replication = 1' for the file, only 1 copy of the
> file's blocks will be stored in Hadoop. If you want all 5 machines to
have
> copies of each block, then you would set 'replication = 5' for the
file.
>
> The default for replication is 3.
>
> Thanks,
> Stu
>
>
--
View this message in context:
http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a125348
39
Sent from the Hadoop Users mailing list archive at Nabble.com.
RE: Replication problem of HDFS
Posted by ChaoChun Liang <cc...@gmail.com>.
So, the upload process(from local file system to HDFS) will store all
blocks(split from the dataset,
said M split blocks) into a single node(depend on which client you put), not
to all datanodes.
And the "replication" means to replicate to N clients(if replication=N) and
each client owns
a completed/all M blocks. If I am wrong, please correct it. Thanks.
ChaoChun
Stu Hood-2 wrote:
>
> ChaoChun,
>
> Since you set the 'replication = 1' for the file, only 1 copy of the
> file's blocks will be stored in Hadoop. If you want all 5 machines to have
> copies of each block, then you would set 'replication = 5' for the file.
>
> The default for replication is 3.
>
> Thanks,
> Stu
>
>
--
View this message in context: http://www.nabble.com/Replication-problem-of-HDFS-tf4382878.html#a12534839
Sent from the Hadoop Users mailing list archive at Nabble.com.