You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Stijn De Weirdt <st...@ugent.be> on 2012/04/02 18:54:24 UTC
data distribution in HDFS
hi all,
i'm just started to play around with hdfs+mapred. i'm currently playing
with teragen/sort/validate to see if i understand all.
the test setup involves 5 nodes that all are tasktracker and datanode
(and one node that is also jobtracker and namenode on top of that. (this
one node is running both the namenode hadoop process as the datanode
process)
when i do the in teragen run, the data is not distributed equally over
all nodes. the node that is also namenode, get's a bigger portion of all
the data. (as seen by df on the nodes and by using dsfadmin -report)
i also get this distribution when i ran the TestDFSIO write test (50
files of 1GB)
i use basic command line teragen $((100*1000*1000))
/benchmarks/teragen, so i expect 100M*0.1kb = 10GB of data. (if i add
the volumes in use by hdfs, it's actually quite a bit more.)
4 data nodes are using 4.2-4.8GB, and the data+namenode has 9.4GB in
use. so this one datanode is seen as 2 nodes.
when i do ls on the filesystem, i see that teragen created 250MB files,
the current hdfs blocksize is 64MB.
is there a reason why one datanode is preferred over the others.
it is annoying since the terasort output behaves the same, and i can't
use the full hdfs space for testing that way. also, since more IO comes
to this one node, the performance isn't really balanced.
many thanks,
stijn
Re: data distribution in HDFS
Posted by Raj Vishwanathan <ra...@yahoo.com>.
AFAIK there is no way to disable this "feature" . This is an optimization. It happens because in your case the node generating the data is also a data node.
Raj
>________________________________
> From: Stijn De Weirdt <st...@ugent.be>
>To: common-user@hadoop.apache.org
>Sent: Monday, April 2, 2012 12:18 PM
>Subject: Re: data distribution in HDFS
>
>thanks serge.
>
>
>is there a way to disable this "feature" (ie place first block always on
>local node)?
>and is this because the local node is a datanode? or is there always a
>"local node" with datatransfers?
>
>many thanks,
>
>stijn
>
>> Local node is a node from where you are coping data from
>>
>> If lets say you are using -copyFromLocal option
>>
>>
>> Regards
>> Serge
>>
>> On 4/2/12 11:53 AM, "Stijn De Weirdt"<st...@ugent.be> wrote:
>>
>>> hi raj,
>>>
>>> what is a "local node"? is it relative to the tasks that are started?
>>>
>>>
>>> stijn
>>>
>>> On 04/02/2012 07:28 PM, Raj Vishwanathan wrote:
>>>> Stijn,
>>>>
>>>> The first block of the data , is always stored in the local node.
>>>> Assuming that you had a replication factor of 3, the node that generates
>>>> the data will get about 10GB of data and the other 20GB will be
>>>> distributed among other nodes.
>>>>
>>>> Raj
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> ________________________________
>>>>> From: Stijn De Weirdt<st...@ugent.be>
>>>>> To: common-user@hadoop.apache.org
>>>>> Sent: Monday, April 2, 2012 9:54 AM
>>>>> Subject: data distribution in HDFS
>>>>>
>>>>> hi all,
>>>>>
>>>>> i'm just started to play around with hdfs+mapred. i'm currently
>>>>> playing with teragen/sort/validate to see if i understand all.
>>>>>
>>>>> the test setup involves 5 nodes that all are tasktracker and datanode
>>>>> (and one node that is also jobtracker and namenode on top of that.
>>>>> (this one node is running both the namenode hadoop process as the
>>>>> datanode process)
>>>>>
>>>>> when i do the in teragen run, the data is not distributed equally over
>>>>> all nodes. the node that is also namenode, get's a bigger portion of
>>>>> all the data. (as seen by df on the nodes and by using dsfadmin -report)
>>>>> i also get this distribution when i ran the TestDFSIO write test (50
>>>>> files of 1GB)
>>>>>
>>>>>
>>>>> i use basic command line teragen $((100*1000*1000))
>>>>> /benchmarks/teragen, so i expect 100M*0.1kb = 10GB of data. (if i add
>>>>> the volumes in use by hdfs, it's actually quite a bit more.)
>>>>> 4 data nodes are using 4.2-4.8GB, and the data+namenode has 9.4GB in
>>>>> use. so this one datanode is seen as 2 nodes.
>>>>>
>>>>> when i do ls on the filesystem, i see that teragen created 250MB
>>>>> files, the current hdfs blocksize is 64MB.
>>>>>
>>>>> is there a reason why one datanode is preferred over the others.
>>>>> it is annoying since the terasort output behaves the same, and i can't
>>>>> use the full hdfs space for testing that way. also, since more IO comes
>>>>> to this one node, the performance isn't really balanced.
>>>>>
>>>>> many thanks,
>>>>>
>>>>> stijn
>>>>>
>>>>>
>>>>>
>>>
>>
>>
>
>
>
>
Re: data distribution in HDFS
Posted by Stijn De Weirdt <st...@ugent.be>.
thanks serge.
is there a way to disable this "feature" (ie place first block always on
local node)?
and is this because the local node is a datanode? or is there always a
"local node" with datatransfers?
many thanks,
stijn
> Local node is a node from where you are coping data from
>
> If lets say you are using -copyFromLocal option
>
>
> Regards
> Serge
>
> On 4/2/12 11:53 AM, "Stijn De Weirdt"<st...@ugent.be> wrote:
>
>> hi raj,
>>
>> what is a "local node"? is it relative to the tasks that are started?
>>
>>
>> stijn
>>
>> On 04/02/2012 07:28 PM, Raj Vishwanathan wrote:
>>> Stijn,
>>>
>>> The first block of the data , is always stored in the local node.
>>> Assuming that you had a replication factor of 3, the node that generates
>>> the data will get about 10GB of data and the other 20GB will be
>>> distributed among other nodes.
>>>
>>> Raj
>>>
>>>
>>>
>>>
>>>
>>>> ________________________________
>>>> From: Stijn De Weirdt<st...@ugent.be>
>>>> To: common-user@hadoop.apache.org
>>>> Sent: Monday, April 2, 2012 9:54 AM
>>>> Subject: data distribution in HDFS
>>>>
>>>> hi all,
>>>>
>>>> i'm just started to play around with hdfs+mapred. i'm currently
>>>> playing with teragen/sort/validate to see if i understand all.
>>>>
>>>> the test setup involves 5 nodes that all are tasktracker and datanode
>>>> (and one node that is also jobtracker and namenode on top of that.
>>>> (this one node is running both the namenode hadoop process as the
>>>> datanode process)
>>>>
>>>> when i do the in teragen run, the data is not distributed equally over
>>>> all nodes. the node that is also namenode, get's a bigger portion of
>>>> all the data. (as seen by df on the nodes and by using dsfadmin -report)
>>>> i also get this distribution when i ran the TestDFSIO write test (50
>>>> files of 1GB)
>>>>
>>>>
>>>> i use basic command line teragen $((100*1000*1000))
>>>> /benchmarks/teragen, so i expect 100M*0.1kb = 10GB of data. (if i add
>>>> the volumes in use by hdfs, it's actually quite a bit more.)
>>>> 4 data nodes are using 4.2-4.8GB, and the data+namenode has 9.4GB in
>>>> use. so this one datanode is seen as 2 nodes.
>>>>
>>>> when i do ls on the filesystem, i see that teragen created 250MB
>>>> files, the current hdfs blocksize is 64MB.
>>>>
>>>> is there a reason why one datanode is preferred over the others.
>>>> it is annoying since the terasort output behaves the same, and i can't
>>>> use the full hdfs space for testing that way. also, since more IO comes
>>>> to this one node, the performance isn't really balanced.
>>>>
>>>> many thanks,
>>>>
>>>> stijn
>>>>
>>>>
>>>>
>>
>
>
Re: data distribution in HDFS
Posted by Serge Blazhievsky <Se...@nice.com>.
Local node is a node from where you are coping data from
If lets say you are using -copyFromLocal option
Regards
Serge
On 4/2/12 11:53 AM, "Stijn De Weirdt" <st...@ugent.be> wrote:
>hi raj,
>
>what is a "local node"? is it relative to the tasks that are started?
>
>
>stijn
>
>On 04/02/2012 07:28 PM, Raj Vishwanathan wrote:
>> Stijn,
>>
>> The first block of the data , is always stored in the local node.
>>Assuming that you had a replication factor of 3, the node that generates
>>the data will get about 10GB of data and the other 20GB will be
>>distributed among other nodes.
>>
>> Raj
>>
>>
>>
>>
>>
>>> ________________________________
>>> From: Stijn De Weirdt<st...@ugent.be>
>>> To: common-user@hadoop.apache.org
>>> Sent: Monday, April 2, 2012 9:54 AM
>>> Subject: data distribution in HDFS
>>>
>>> hi all,
>>>
>>> i'm just started to play around with hdfs+mapred. i'm currently
>>>playing with teragen/sort/validate to see if i understand all.
>>>
>>> the test setup involves 5 nodes that all are tasktracker and datanode
>>>(and one node that is also jobtracker and namenode on top of that.
>>>(this one node is running both the namenode hadoop process as the
>>>datanode process)
>>>
>>> when i do the in teragen run, the data is not distributed equally over
>>>all nodes. the node that is also namenode, get's a bigger portion of
>>>all the data. (as seen by df on the nodes and by using dsfadmin -report)
>>> i also get this distribution when i ran the TestDFSIO write test (50
>>>files of 1GB)
>>>
>>>
>>> i use basic command line teragen $((100*1000*1000))
>>>/benchmarks/teragen, so i expect 100M*0.1kb = 10GB of data. (if i add
>>>the volumes in use by hdfs, it's actually quite a bit more.)
>>> 4 data nodes are using 4.2-4.8GB, and the data+namenode has 9.4GB in
>>>use. so this one datanode is seen as 2 nodes.
>>>
>>> when i do ls on the filesystem, i see that teragen created 250MB
>>>files, the current hdfs blocksize is 64MB.
>>>
>>> is there a reason why one datanode is preferred over the others.
>>> it is annoying since the terasort output behaves the same, and i can't
>>>use the full hdfs space for testing that way. also, since more IO comes
>>>to this one node, the performance isn't really balanced.
>>>
>>> many thanks,
>>>
>>> stijn
>>>
>>>
>>>
>
Re: data distribution in HDFS
Posted by Stijn De Weirdt <st...@ugent.be>.
hi raj,
what is a "local node"? is it relative to the tasks that are started?
stijn
On 04/02/2012 07:28 PM, Raj Vishwanathan wrote:
> Stijn,
>
> The first block of the data , is always stored in the local node. Assuming that you had a replication factor of 3, the node that generates the data will get about 10GB of data and the other 20GB will be distributed among other nodes.
>
> Raj
>
>
>
>
>
>> ________________________________
>> From: Stijn De Weirdt<st...@ugent.be>
>> To: common-user@hadoop.apache.org
>> Sent: Monday, April 2, 2012 9:54 AM
>> Subject: data distribution in HDFS
>>
>> hi all,
>>
>> i'm just started to play around with hdfs+mapred. i'm currently playing with teragen/sort/validate to see if i understand all.
>>
>> the test setup involves 5 nodes that all are tasktracker and datanode (and one node that is also jobtracker and namenode on top of that. (this one node is running both the namenode hadoop process as the datanode process)
>>
>> when i do the in teragen run, the data is not distributed equally over all nodes. the node that is also namenode, get's a bigger portion of all the data. (as seen by df on the nodes and by using dsfadmin -report)
>> i also get this distribution when i ran the TestDFSIO write test (50 files of 1GB)
>>
>>
>> i use basic command line teragen $((100*1000*1000)) /benchmarks/teragen, so i expect 100M*0.1kb = 10GB of data. (if i add the volumes in use by hdfs, it's actually quite a bit more.)
>> 4 data nodes are using 4.2-4.8GB, and the data+namenode has 9.4GB in use. so this one datanode is seen as 2 nodes.
>>
>> when i do ls on the filesystem, i see that teragen created 250MB files, the current hdfs blocksize is 64MB.
>>
>> is there a reason why one datanode is preferred over the others.
>> it is annoying since the terasort output behaves the same, and i can't use the full hdfs space for testing that way. also, since more IO comes to this one node, the performance isn't really balanced.
>>
>> many thanks,
>>
>> stijn
>>
>>
>>
Re: data distribution in HDFS
Posted by Raj Vishwanathan <ra...@yahoo.com>.
Stijn,
The first block of the data , is always stored in the local node. Assuming that you had a replication factor of 3, the node that generates the data will get about 10GB of data and the other 20GB will be distributed among other nodes.
Raj
>________________________________
> From: Stijn De Weirdt <st...@ugent.be>
>To: common-user@hadoop.apache.org
>Sent: Monday, April 2, 2012 9:54 AM
>Subject: data distribution in HDFS
>
>hi all,
>
>i'm just started to play around with hdfs+mapred. i'm currently playing with teragen/sort/validate to see if i understand all.
>
>the test setup involves 5 nodes that all are tasktracker and datanode (and one node that is also jobtracker and namenode on top of that. (this one node is running both the namenode hadoop process as the datanode process)
>
>when i do the in teragen run, the data is not distributed equally over all nodes. the node that is also namenode, get's a bigger portion of all the data. (as seen by df on the nodes and by using dsfadmin -report)
>i also get this distribution when i ran the TestDFSIO write test (50 files of 1GB)
>
>
>i use basic command line teragen $((100*1000*1000)) /benchmarks/teragen, so i expect 100M*0.1kb = 10GB of data. (if i add the volumes in use by hdfs, it's actually quite a bit more.)
>4 data nodes are using 4.2-4.8GB, and the data+namenode has 9.4GB in use. so this one datanode is seen as 2 nodes.
>
>when i do ls on the filesystem, i see that teragen created 250MB files, the current hdfs blocksize is 64MB.
>
>is there a reason why one datanode is preferred over the others.
>it is annoying since the terasort output behaves the same, and i can't use the full hdfs space for testing that way. also, since more IO comes to this one node, the performance isn't really balanced.
>
>many thanks,
>
>stijn
>
>
>