You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Marcos Sousa <fa...@marcossousa.com> on 2014/05/12 20:52:19 UTC
Data node with multiple disks
Hi,
I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be my
datanode:
/vol1/hadoop/data
/vol2/hadoop/data
/vol3/hadoop/data
/volN/hadoop/data
How do user those distinct discs not to replicate?
Best regards,
--
Marcos Sousa
Re: Data node with multiple disks
Posted by Nitin Pawar <ni...@gmail.com>.
Hi Marcos,
If these discs are not shared across nodes, I would not worry. Hadoop takes
care of making sure data is not replicated to single node.
But if all these 20 nodes are sharing these 10 HDD's,
Then you may have to basically assign specific disc to specific node and
make your cluster rack aware so that the replication in same rack would go
to different node and replication to second rack will to new disc.
On Tue, May 13, 2014 at 1:38 PM, kishore alajangi <alajangikishore@gmail.com
> wrote:
> replication factor=1
>
>
> On Tue, May 13, 2014 at 11:04 AM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Your question is unclear. Please restate and describe what you are
>> attempting to do.
>>
>> Thanks.
>>
>>
>> On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
>>
>>> Hi,
>>>
>>> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
>>> my datanode:
>>>
>>> /vol1/hadoop/data
>>> /vol2/hadoop/data
>>> /vol3/hadoop/data
>>> /volN/hadoop/data
>>>
>>> How do user those distinct discs not to replicate?
>>>
>>> Best regards,
>>>
>>> --
>>> Marcos Sousa
>>>
>>
>
>
> --
> Thanks,
> Kishore.
>
--
Nitin Pawar
Re: Data node with multiple disks
Posted by Nitin Pawar <ni...@gmail.com>.
Hi Marcos,
If these discs are not shared across nodes, I would not worry. Hadoop takes
care of making sure data is not replicated to single node.
But if all these 20 nodes are sharing these 10 HDD's,
Then you may have to basically assign specific disc to specific node and
make your cluster rack aware so that the replication in same rack would go
to different node and replication to second rack will to new disc.
On Tue, May 13, 2014 at 1:38 PM, kishore alajangi <alajangikishore@gmail.com
> wrote:
> replication factor=1
>
>
> On Tue, May 13, 2014 at 11:04 AM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Your question is unclear. Please restate and describe what you are
>> attempting to do.
>>
>> Thanks.
>>
>>
>> On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
>>
>>> Hi,
>>>
>>> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
>>> my datanode:
>>>
>>> /vol1/hadoop/data
>>> /vol2/hadoop/data
>>> /vol3/hadoop/data
>>> /volN/hadoop/data
>>>
>>> How do user those distinct discs not to replicate?
>>>
>>> Best regards,
>>>
>>> --
>>> Marcos Sousa
>>>
>>
>
>
> --
> Thanks,
> Kishore.
>
--
Nitin Pawar
Re: Data node with multiple disks
Posted by Nitin Pawar <ni...@gmail.com>.
Hi Marcos,
If these discs are not shared across nodes, I would not worry. Hadoop takes
care of making sure data is not replicated to single node.
But if all these 20 nodes are sharing these 10 HDD's,
Then you may have to basically assign specific disc to specific node and
make your cluster rack aware so that the replication in same rack would go
to different node and replication to second rack will to new disc.
On Tue, May 13, 2014 at 1:38 PM, kishore alajangi <alajangikishore@gmail.com
> wrote:
> replication factor=1
>
>
> On Tue, May 13, 2014 at 11:04 AM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Your question is unclear. Please restate and describe what you are
>> attempting to do.
>>
>> Thanks.
>>
>>
>> On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
>>
>>> Hi,
>>>
>>> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
>>> my datanode:
>>>
>>> /vol1/hadoop/data
>>> /vol2/hadoop/data
>>> /vol3/hadoop/data
>>> /volN/hadoop/data
>>>
>>> How do user those distinct discs not to replicate?
>>>
>>> Best regards,
>>>
>>> --
>>> Marcos Sousa
>>>
>>
>
>
> --
> Thanks,
> Kishore.
>
--
Nitin Pawar
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by Harsh J <ha...@cloudera.com>.
You will need to set them up in the hdfs-site.xml.
P.s. Their default is present in the hdfs-default.xml you linked to:
http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml#dfs.datanode.data.dir
On Sat, Aug 2, 2014 at 12:29 PM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> Both ”dfs.name.data.dir” and “dfs.datanode.data.dir” are not set in my
> cluster. By the way I have searched around about these two parameters, I
> cannot find them in Hadoop Default page.
> http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
>
> Can you please advise where to set them and how to set them? in
> hdfs-site.xml or in core-site.xml or another configuration file?
>
> Many thanks
> Arthur
>
> On 29 Jul, 2014, at 1:27 am, hadoop hive <ha...@gmail.com> wrote:
>
> You need to add each disk inside "dfs.name.data.dir" parameter.
>
> On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com"
> <ar...@gmail.com> wrote:
>>
>> Hi,
>>
>> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T
>> hard disk, when checking the configured capacity, I found it is about 49.22
>> GB per node, can anyone advise how to set bigger “configured capacity” e.g.
>> 2T or more per node?
>>
>> Name node
>> Configured Capacity: 264223436800 (246.08 GB)
>>
>> Each Datanode
>> Configured Capacity: 52844687360 (49.22 GB)
>>
>> regards
>> Arthur
>
>
--
Harsh J
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by Harsh J <ha...@cloudera.com>.
You will need to set them up in the hdfs-site.xml.
P.s. Their default is present in the hdfs-default.xml you linked to:
http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml#dfs.datanode.data.dir
On Sat, Aug 2, 2014 at 12:29 PM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> Both ”dfs.name.data.dir” and “dfs.datanode.data.dir” are not set in my
> cluster. By the way I have searched around about these two parameters, I
> cannot find them in Hadoop Default page.
> http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
>
> Can you please advise where to set them and how to set them? in
> hdfs-site.xml or in core-site.xml or another configuration file?
>
> Many thanks
> Arthur
>
> On 29 Jul, 2014, at 1:27 am, hadoop hive <ha...@gmail.com> wrote:
>
> You need to add each disk inside "dfs.name.data.dir" parameter.
>
> On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com"
> <ar...@gmail.com> wrote:
>>
>> Hi,
>>
>> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T
>> hard disk, when checking the configured capacity, I found it is about 49.22
>> GB per node, can anyone advise how to set bigger “configured capacity” e.g.
>> 2T or more per node?
>>
>> Name node
>> Configured Capacity: 264223436800 (246.08 GB)
>>
>> Each Datanode
>> Configured Capacity: 52844687360 (49.22 GB)
>>
>> regards
>> Arthur
>
>
--
Harsh J
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by Harsh J <ha...@cloudera.com>.
You will need to set them up in the hdfs-site.xml.
P.s. Their default is present in the hdfs-default.xml you linked to:
http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml#dfs.datanode.data.dir
On Sat, Aug 2, 2014 at 12:29 PM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> Both ”dfs.name.data.dir” and “dfs.datanode.data.dir” are not set in my
> cluster. By the way I have searched around about these two parameters, I
> cannot find them in Hadoop Default page.
> http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
>
> Can you please advise where to set them and how to set them? in
> hdfs-site.xml or in core-site.xml or another configuration file?
>
> Many thanks
> Arthur
>
> On 29 Jul, 2014, at 1:27 am, hadoop hive <ha...@gmail.com> wrote:
>
> You need to add each disk inside "dfs.name.data.dir" parameter.
>
> On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com"
> <ar...@gmail.com> wrote:
>>
>> Hi,
>>
>> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T
>> hard disk, when checking the configured capacity, I found it is about 49.22
>> GB per node, can anyone advise how to set bigger “configured capacity” e.g.
>> 2T or more per node?
>>
>> Name node
>> Configured Capacity: 264223436800 (246.08 GB)
>>
>> Each Datanode
>> Configured Capacity: 52844687360 (49.22 GB)
>>
>> regards
>> Arthur
>
>
--
Harsh J
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by Harsh J <ha...@cloudera.com>.
You will need to set them up in the hdfs-site.xml.
P.s. Their default is present in the hdfs-default.xml you linked to:
http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml#dfs.datanode.data.dir
On Sat, Aug 2, 2014 at 12:29 PM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> Both ”dfs.name.data.dir” and “dfs.datanode.data.dir” are not set in my
> cluster. By the way I have searched around about these two parameters, I
> cannot find them in Hadoop Default page.
> http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
>
> Can you please advise where to set them and how to set them? in
> hdfs-site.xml or in core-site.xml or another configuration file?
>
> Many thanks
> Arthur
>
> On 29 Jul, 2014, at 1:27 am, hadoop hive <ha...@gmail.com> wrote:
>
> You need to add each disk inside "dfs.name.data.dir" parameter.
>
> On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com"
> <ar...@gmail.com> wrote:
>>
>> Hi,
>>
>> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T
>> hard disk, when checking the configured capacity, I found it is about 49.22
>> GB per node, can anyone advise how to set bigger “configured capacity” e.g.
>> 2T or more per node?
>>
>> Name node
>> Configured Capacity: 264223436800 (246.08 GB)
>>
>> Each Datanode
>> Configured Capacity: 52844687360 (49.22 GB)
>>
>> regards
>> Arthur
>
>
--
Harsh J
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
Both ”dfs.name.data.dir” and “dfs.datanode.data.dir” are not set in my cluster. By the way I have searched around about these two parameters, I cannot find them in Hadoop Default page.
http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Can you please advise where to set them and how to set them? in hdfs-site.xml or in core-site.xml or another configuration file?
Many thanks
Arthur
On 29 Jul, 2014, at 1:27 am, hadoop hive <ha...@gmail.com> wrote:
> You need to add each disk inside "dfs.name.data.dir" parameter.
>
> On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com" <ar...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard disk, when checking the configured capacity, I found it is about 49.22 GB per node, can anyone advise how to set bigger “configured capacity” e.g. 2T or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
Both ”dfs.name.data.dir” and “dfs.datanode.data.dir” are not set in my cluster. By the way I have searched around about these two parameters, I cannot find them in Hadoop Default page.
http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Can you please advise where to set them and how to set them? in hdfs-site.xml or in core-site.xml or another configuration file?
Many thanks
Arthur
On 29 Jul, 2014, at 1:27 am, hadoop hive <ha...@gmail.com> wrote:
> You need to add each disk inside "dfs.name.data.dir" parameter.
>
> On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com" <ar...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard disk, when checking the configured capacity, I found it is about 49.22 GB per node, can anyone advise how to set bigger “configured capacity” e.g. 2T or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
Both ”dfs.name.data.dir” and “dfs.datanode.data.dir” are not set in my cluster. By the way I have searched around about these two parameters, I cannot find them in Hadoop Default page.
http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Can you please advise where to set them and how to set them? in hdfs-site.xml or in core-site.xml or another configuration file?
Many thanks
Arthur
On 29 Jul, 2014, at 1:27 am, hadoop hive <ha...@gmail.com> wrote:
> You need to add each disk inside "dfs.name.data.dir" parameter.
>
> On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com" <ar...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard disk, when checking the configured capacity, I found it is about 49.22 GB per node, can anyone advise how to set bigger “configured capacity” e.g. 2T or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
Both ”dfs.name.data.dir” and “dfs.datanode.data.dir” are not set in my cluster. By the way I have searched around about these two parameters, I cannot find them in Hadoop Default page.
http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Can you please advise where to set them and how to set them? in hdfs-site.xml or in core-site.xml or another configuration file?
Many thanks
Arthur
On 29 Jul, 2014, at 1:27 am, hadoop hive <ha...@gmail.com> wrote:
> You need to add each disk inside "dfs.name.data.dir" parameter.
>
> On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com" <ar...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard disk, when checking the configured capacity, I found it is about 49.22 GB per node, can anyone advise how to set bigger “configured capacity” e.g. 2T or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by hadoop hive <ha...@gmail.com>.
You need to add each disk inside "dfs.name.data.dir" parameter.
On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com" <
arthur.hk.chan@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T
> hard disk, when checking the configured capacity, I found it is about 49.22
> GB per node, can anyone advise how to set bigger “configured capacity” e.g.
> 2T or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
>
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by hadoop hive <ha...@gmail.com>.
You need to add each disk inside "dfs.name.data.dir" parameter.
On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com" <
arthur.hk.chan@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T
> hard disk, when checking the configured capacity, I found it is about 49.22
> GB per node, can anyone advise how to set bigger “configured capacity” e.g.
> 2T or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
>
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by hadoop hive <ha...@gmail.com>.
You need to add each disk inside "dfs.name.data.dir" parameter.
On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com" <
arthur.hk.chan@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T
> hard disk, when checking the configured capacity, I found it is about 49.22
> GB per node, can anyone advise how to set bigger “configured capacity” e.g.
> 2T or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
>
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
Did you configure the property "dfs.datanode.data.dir" with multiple
mount point?
Thanks,
- Tsuyoshi
On Mon, Jul 28, 2014 at 8:43 AM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard
> disk, when checking the configured capacity, I found it is about 49.22 GB
> per node, can anyone advise how to set bigger “configured capacity” e.g. 2T
> or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
--
- Tsuyoshi
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
Did you configure the property "dfs.datanode.data.dir" with multiple
mount point?
Thanks,
- Tsuyoshi
On Mon, Jul 28, 2014 at 8:43 AM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard
> disk, when checking the configured capacity, I found it is about 49.22 GB
> per node, can anyone advise how to set bigger “configured capacity” e.g. 2T
> or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
--
- Tsuyoshi
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
Did you configure the property "dfs.datanode.data.dir" with multiple
mount point?
Thanks,
- Tsuyoshi
On Mon, Jul 28, 2014 at 8:43 AM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard
> disk, when checking the configured capacity, I found it is about 49.22 GB
> per node, can anyone advise how to set bigger “configured capacity” e.g. 2T
> or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
--
- Tsuyoshi
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by hadoop hive <ha...@gmail.com>.
You need to add each disk inside "dfs.name.data.dir" parameter.
On Jul 28, 2014 5:14 AM, "Arthur.hk.chan@gmail.com" <
arthur.hk.chan@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T
> hard disk, when checking the configured capacity, I found it is about 49.22
> GB per node, can anyone advise how to set bigger “configured capacity” e.g.
> 2T or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
>
Re: Hadoop 2.4.0 How to change "Configured Capacity"
Posted by Tsuyoshi OZAWA <oz...@gmail.com>.
Hi,
Did you configure the property "dfs.datanode.data.dir" with multiple
mount point?
Thanks,
- Tsuyoshi
On Mon, Jul 28, 2014 at 8:43 AM, Arthur.hk.chan@gmail.com
<ar...@gmail.com> wrote:
> Hi,
>
> I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard
> disk, when checking the configured capacity, I found it is about 49.22 GB
> per node, can anyone advise how to set bigger “configured capacity” e.g. 2T
> or more per node?
>
> Name node
> Configured Capacity: 264223436800 (246.08 GB)
>
> Each Datanode
> Configured Capacity: 52844687360 (49.22 GB)
>
> regards
> Arthur
--
- Tsuyoshi
Hadoop 2.4.0 How to change "Configured Capacity"
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard disk, when checking the configured capacity, I found it is about 49.22 GB per node, can anyone advise how to set bigger “configured capacity” e.g. 2T or more per node?
Name node
Configured Capacity: 264223436800 (246.08 GB)
Each Datanode
Configured Capacity: 52844687360 (49.22 GB)
regards
Arthur
Hadoop 2.4.0 How to change "Configured Capacity"
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard disk, when checking the configured capacity, I found it is about 49.22 GB per node, can anyone advise how to set bigger “configured capacity” e.g. 2T or more per node?
Name node
Configured Capacity: 264223436800 (246.08 GB)
Each Datanode
Configured Capacity: 52844687360 (49.22 GB)
regards
Arthur
Re: Data node with multiple disks
Posted by Nitin Pawar <ni...@gmail.com>.
Hi Marcos,
If these discs are not shared across nodes, I would not worry. Hadoop takes
care of making sure data is not replicated to single node.
But if all these 20 nodes are sharing these 10 HDD's,
Then you may have to basically assign specific disc to specific node and
make your cluster rack aware so that the replication in same rack would go
to different node and replication to second rack will to new disc.
On Tue, May 13, 2014 at 1:38 PM, kishore alajangi <alajangikishore@gmail.com
> wrote:
> replication factor=1
>
>
> On Tue, May 13, 2014 at 11:04 AM, SF Hadoop <sf...@gmail.com> wrote:
>
>> Your question is unclear. Please restate and describe what you are
>> attempting to do.
>>
>> Thanks.
>>
>>
>> On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
>>
>>> Hi,
>>>
>>> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
>>> my datanode:
>>>
>>> /vol1/hadoop/data
>>> /vol2/hadoop/data
>>> /vol3/hadoop/data
>>> /volN/hadoop/data
>>>
>>> How do user those distinct discs not to replicate?
>>>
>>> Best regards,
>>>
>>> --
>>> Marcos Sousa
>>>
>>
>
>
> --
> Thanks,
> Kishore.
>
--
Nitin Pawar
Hadoop 2.4.0 How to change "Configured Capacity"
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard disk, when checking the configured capacity, I found it is about 49.22 GB per node, can anyone advise how to set bigger “configured capacity” e.g. 2T or more per node?
Name node
Configured Capacity: 264223436800 (246.08 GB)
Each Datanode
Configured Capacity: 52844687360 (49.22 GB)
regards
Arthur
Hadoop 2.4.0 How to change "Configured Capacity"
Posted by "Arthur.hk.chan@gmail.com" <ar...@gmail.com>.
Hi,
I have installed Hadoop 2.4.0 with 5 nodes, each node physically has 4T hard disk, when checking the configured capacity, I found it is about 49.22 GB per node, can anyone advise how to set bigger “configured capacity” e.g. 2T or more per node?
Name node
Configured Capacity: 264223436800 (246.08 GB)
Each Datanode
Configured Capacity: 52844687360 (49.22 GB)
regards
Arthur
Re: Data node with multiple disks
Posted by kishore alajangi <al...@gmail.com>.
replication factor=1
On Tue, May 13, 2014 at 11:04 AM, SF Hadoop <sf...@gmail.com> wrote:
> Your question is unclear. Please restate and describe what you are
> attempting to do.
>
> Thanks.
>
>
> On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
>
>> Hi,
>>
>> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
>> my datanode:
>>
>> /vol1/hadoop/data
>> /vol2/hadoop/data
>> /vol3/hadoop/data
>> /volN/hadoop/data
>>
>> How do user those distinct discs not to replicate?
>>
>> Best regards,
>>
>> --
>> Marcos Sousa
>>
>
--
Thanks,
Kishore.
Re: Data node with multiple disks
Posted by kishore alajangi <al...@gmail.com>.
replication factor=1
On Tue, May 13, 2014 at 11:04 AM, SF Hadoop <sf...@gmail.com> wrote:
> Your question is unclear. Please restate and describe what you are
> attempting to do.
>
> Thanks.
>
>
> On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
>
>> Hi,
>>
>> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
>> my datanode:
>>
>> /vol1/hadoop/data
>> /vol2/hadoop/data
>> /vol3/hadoop/data
>> /volN/hadoop/data
>>
>> How do user those distinct discs not to replicate?
>>
>> Best regards,
>>
>> --
>> Marcos Sousa
>>
>
--
Thanks,
Kishore.
Re: Data node with multiple disks
Posted by kishore alajangi <al...@gmail.com>.
replication factor=1
On Tue, May 13, 2014 at 11:04 AM, SF Hadoop <sf...@gmail.com> wrote:
> Your question is unclear. Please restate and describe what you are
> attempting to do.
>
> Thanks.
>
>
> On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
>
>> Hi,
>>
>> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
>> my datanode:
>>
>> /vol1/hadoop/data
>> /vol2/hadoop/data
>> /vol3/hadoop/data
>> /volN/hadoop/data
>>
>> How do user those distinct discs not to replicate?
>>
>> Best regards,
>>
>> --
>> Marcos Sousa
>>
>
--
Thanks,
Kishore.
Re: Data node with multiple disks
Posted by kishore alajangi <al...@gmail.com>.
replication factor=1
On Tue, May 13, 2014 at 11:04 AM, SF Hadoop <sf...@gmail.com> wrote:
> Your question is unclear. Please restate and describe what you are
> attempting to do.
>
> Thanks.
>
>
> On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
>
>> Hi,
>>
>> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
>> my datanode:
>>
>> /vol1/hadoop/data
>> /vol2/hadoop/data
>> /vol3/hadoop/data
>> /volN/hadoop/data
>>
>> How do user those distinct discs not to replicate?
>>
>> Best regards,
>>
>> --
>> Marcos Sousa
>>
>
--
Thanks,
Kishore.
Re: Data node with multiple disks
Posted by SF Hadoop <sf...@gmail.com>.
Your question is unclear. Please restate and describe what you are
attempting to do.
Thanks.
On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
> Hi,
>
> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
> my datanode:
>
> /vol1/hadoop/data
> /vol2/hadoop/data
> /vol3/hadoop/data
> /volN/hadoop/data
>
> How do user those distinct discs not to replicate?
>
> Best regards,
>
> --
> Marcos Sousa
>
Re: Data node with multiple disks
Posted by SF Hadoop <sf...@gmail.com>.
Your question is unclear. Please restate and describe what you are
attempting to do.
Thanks.
On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
> Hi,
>
> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
> my datanode:
>
> /vol1/hadoop/data
> /vol2/hadoop/data
> /vol3/hadoop/data
> /volN/hadoop/data
>
> How do user those distinct discs not to replicate?
>
> Best regards,
>
> --
> Marcos Sousa
>
Re: Data node with multiple disks
Posted by SF Hadoop <sf...@gmail.com>.
Your question is unclear. Please restate and describe what you are
attempting to do.
Thanks.
On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
> Hi,
>
> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
> my datanode:
>
> /vol1/hadoop/data
> /vol2/hadoop/data
> /vol3/hadoop/data
> /volN/hadoop/data
>
> How do user those distinct discs not to replicate?
>
> Best regards,
>
> --
> Marcos Sousa
>
Re: Data node with multiple disks
Posted by Aitor Perez Cedres <ap...@pragsis.com>.
If you specify a list in the property "dfs.datanode.data.dir" hadoop
will distribute the data blocks among all those disks; it will not
replicate data between them. If you want to use the disks as a single
one you gotta make a LVM array or any other solution to present them as
a single one to the OS.
However, benchmarks prove that specifying a list of disks and letting
hadoop distribute data among them gives better performance.
On 13/05/14 17:12, Marcos Sousa wrote:
> Yes,
>
> I don't want to replicate, just use as one disk? Isn't possible to
> make this work?
>
> Best regards,
>
> Marcos
>
>
> On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari
> <rahulchaudhari0405@gmail.com <ma...@gmail.com>>
> wrote:
>
> Marcos,
> While configuring hadoop, the "dfs.datanode.data.dir" property
> in hdfs-default.xml should have this list of disks specified on
> separate line. If you specific comma separated list, it will
> replicate on all those disks/partitions.
>
> _Rahul
> Sent from my iPad
>
> > On 13-May-2014, at 12:22 am, Marcos Sousa
> <falecomigo@marcossousa.com <ma...@marcossousa.com>>
> wrote:
> >
> > Hi,
> >
> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use
> them to be my datanode:
> >
> > /vol1/hadoop/data
> > /vol2/hadoop/data
> > /vol3/hadoop/data
> > /volN/hadoop/data
> >
> > How do user those distinct discs not to replicate?
> >
> > Best regards,
> >
> > --
> > Marcos Sousa
>
>
>
>
> --
> Marcos Sousa
> www.marcossousa.com <http://www.marcossousa.com> Enjoy it!
--
*Aitor Pérez*
/Big Data System Engineer/
Telf.: +34 917 680 490
Fax: +34 913 833 301
C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
_http://www.bidoop.es_
Re: Data node with multiple disks
Posted by SF Hadoop <sf...@gmail.com>.
just set you replication factor to 1 and you will be fine.
On Tue, May 13, 2014 at 8:12 AM, Marcos Sousa <fa...@marcossousa.com>wrote:
> Yes,
>
> I don't want to replicate, just use as one disk? Isn't possible to make
> this work?
>
> Best regards,
>
> Marcos
>
>
> On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari <
> rahulchaudhari0405@gmail.com> wrote:
>
>> Marcos,
>> While configuring hadoop, the "dfs.datanode.data.dir" property in
>> hdfs-default.xml should have this list of disks specified on separate line.
>> If you specific comma separated list, it will replicate on all those
>> disks/partitions.
>>
>> _Rahul
>> Sent from my iPad
>>
>> > On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to
>> be my datanode:
>> >
>> > /vol1/hadoop/data
>> > /vol2/hadoop/data
>> > /vol3/hadoop/data
>> > /volN/hadoop/data
>> >
>> > How do user those distinct discs not to replicate?
>> >
>> > Best regards,
>> >
>> > --
>> > Marcos Sousa
>>
>
>
>
> --
> Marcos Sousa
> www.marcossousa.com Enjoy it!
>
Re: Data node with multiple disks
Posted by Aitor Perez Cedres <ap...@pragsis.com>.
If you specify a list in the property "dfs.datanode.data.dir" hadoop
will distribute the data blocks among all those disks; it will not
replicate data between them. If you want to use the disks as a single
one you gotta make a LVM array or any other solution to present them as
a single one to the OS.
However, benchmarks prove that specifying a list of disks and letting
hadoop distribute data among them gives better performance.
On 13/05/14 17:12, Marcos Sousa wrote:
> Yes,
>
> I don't want to replicate, just use as one disk? Isn't possible to
> make this work?
>
> Best regards,
>
> Marcos
>
>
> On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari
> <rahulchaudhari0405@gmail.com <ma...@gmail.com>>
> wrote:
>
> Marcos,
> While configuring hadoop, the "dfs.datanode.data.dir" property
> in hdfs-default.xml should have this list of disks specified on
> separate line. If you specific comma separated list, it will
> replicate on all those disks/partitions.
>
> _Rahul
> Sent from my iPad
>
> > On 13-May-2014, at 12:22 am, Marcos Sousa
> <falecomigo@marcossousa.com <ma...@marcossousa.com>>
> wrote:
> >
> > Hi,
> >
> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use
> them to be my datanode:
> >
> > /vol1/hadoop/data
> > /vol2/hadoop/data
> > /vol3/hadoop/data
> > /volN/hadoop/data
> >
> > How do user those distinct discs not to replicate?
> >
> > Best regards,
> >
> > --
> > Marcos Sousa
>
>
>
>
> --
> Marcos Sousa
> www.marcossousa.com <http://www.marcossousa.com> Enjoy it!
--
*Aitor Pérez*
/Big Data System Engineer/
Telf.: +34 917 680 490
Fax: +34 913 833 301
C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
_http://www.bidoop.es_
Re: Data node with multiple disks
Posted by Aitor Perez Cedres <ap...@pragsis.com>.
If you specify a list in the property "dfs.datanode.data.dir" hadoop
will distribute the data blocks among all those disks; it will not
replicate data between them. If you want to use the disks as a single
one you gotta make a LVM array or any other solution to present them as
a single one to the OS.
However, benchmarks prove that specifying a list of disks and letting
hadoop distribute data among them gives better performance.
On 13/05/14 17:12, Marcos Sousa wrote:
> Yes,
>
> I don't want to replicate, just use as one disk? Isn't possible to
> make this work?
>
> Best regards,
>
> Marcos
>
>
> On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari
> <rahulchaudhari0405@gmail.com <ma...@gmail.com>>
> wrote:
>
> Marcos,
> While configuring hadoop, the "dfs.datanode.data.dir" property
> in hdfs-default.xml should have this list of disks specified on
> separate line. If you specific comma separated list, it will
> replicate on all those disks/partitions.
>
> _Rahul
> Sent from my iPad
>
> > On 13-May-2014, at 12:22 am, Marcos Sousa
> <falecomigo@marcossousa.com <ma...@marcossousa.com>>
> wrote:
> >
> > Hi,
> >
> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use
> them to be my datanode:
> >
> > /vol1/hadoop/data
> > /vol2/hadoop/data
> > /vol3/hadoop/data
> > /volN/hadoop/data
> >
> > How do user those distinct discs not to replicate?
> >
> > Best regards,
> >
> > --
> > Marcos Sousa
>
>
>
>
> --
> Marcos Sousa
> www.marcossousa.com <http://www.marcossousa.com> Enjoy it!
--
*Aitor Pérez*
/Big Data System Engineer/
Telf.: +34 917 680 490
Fax: +34 913 833 301
C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
_http://www.bidoop.es_
Re: Data node with multiple disks
Posted by SF Hadoop <sf...@gmail.com>.
just set you replication factor to 1 and you will be fine.
On Tue, May 13, 2014 at 8:12 AM, Marcos Sousa <fa...@marcossousa.com>wrote:
> Yes,
>
> I don't want to replicate, just use as one disk? Isn't possible to make
> this work?
>
> Best regards,
>
> Marcos
>
>
> On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari <
> rahulchaudhari0405@gmail.com> wrote:
>
>> Marcos,
>> While configuring hadoop, the "dfs.datanode.data.dir" property in
>> hdfs-default.xml should have this list of disks specified on separate line.
>> If you specific comma separated list, it will replicate on all those
>> disks/partitions.
>>
>> _Rahul
>> Sent from my iPad
>>
>> > On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to
>> be my datanode:
>> >
>> > /vol1/hadoop/data
>> > /vol2/hadoop/data
>> > /vol3/hadoop/data
>> > /volN/hadoop/data
>> >
>> > How do user those distinct discs not to replicate?
>> >
>> > Best regards,
>> >
>> > --
>> > Marcos Sousa
>>
>
>
>
> --
> Marcos Sousa
> www.marcossousa.com Enjoy it!
>
Re: Data node with multiple disks
Posted by SF Hadoop <sf...@gmail.com>.
just set you replication factor to 1 and you will be fine.
On Tue, May 13, 2014 at 8:12 AM, Marcos Sousa <fa...@marcossousa.com>wrote:
> Yes,
>
> I don't want to replicate, just use as one disk? Isn't possible to make
> this work?
>
> Best regards,
>
> Marcos
>
>
> On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari <
> rahulchaudhari0405@gmail.com> wrote:
>
>> Marcos,
>> While configuring hadoop, the "dfs.datanode.data.dir" property in
>> hdfs-default.xml should have this list of disks specified on separate line.
>> If you specific comma separated list, it will replicate on all those
>> disks/partitions.
>>
>> _Rahul
>> Sent from my iPad
>>
>> > On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to
>> be my datanode:
>> >
>> > /vol1/hadoop/data
>> > /vol2/hadoop/data
>> > /vol3/hadoop/data
>> > /volN/hadoop/data
>> >
>> > How do user those distinct discs not to replicate?
>> >
>> > Best regards,
>> >
>> > --
>> > Marcos Sousa
>>
>
>
>
> --
> Marcos Sousa
> www.marcossousa.com Enjoy it!
>
Re: Data node with multiple disks
Posted by SF Hadoop <sf...@gmail.com>.
just set you replication factor to 1 and you will be fine.
On Tue, May 13, 2014 at 8:12 AM, Marcos Sousa <fa...@marcossousa.com>wrote:
> Yes,
>
> I don't want to replicate, just use as one disk? Isn't possible to make
> this work?
>
> Best regards,
>
> Marcos
>
>
> On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari <
> rahulchaudhari0405@gmail.com> wrote:
>
>> Marcos,
>> While configuring hadoop, the "dfs.datanode.data.dir" property in
>> hdfs-default.xml should have this list of disks specified on separate line.
>> If you specific comma separated list, it will replicate on all those
>> disks/partitions.
>>
>> _Rahul
>> Sent from my iPad
>>
>> > On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com>
>> wrote:
>> >
>> > Hi,
>> >
>> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to
>> be my datanode:
>> >
>> > /vol1/hadoop/data
>> > /vol2/hadoop/data
>> > /vol3/hadoop/data
>> > /volN/hadoop/data
>> >
>> > How do user those distinct discs not to replicate?
>> >
>> > Best regards,
>> >
>> > --
>> > Marcos Sousa
>>
>
>
>
> --
> Marcos Sousa
> www.marcossousa.com Enjoy it!
>
Re: Data node with multiple disks
Posted by Aitor Perez Cedres <ap...@pragsis.com>.
If you specify a list in the property "dfs.datanode.data.dir" hadoop
will distribute the data blocks among all those disks; it will not
replicate data between them. If you want to use the disks as a single
one you gotta make a LVM array or any other solution to present them as
a single one to the OS.
However, benchmarks prove that specifying a list of disks and letting
hadoop distribute data among them gives better performance.
On 13/05/14 17:12, Marcos Sousa wrote:
> Yes,
>
> I don't want to replicate, just use as one disk? Isn't possible to
> make this work?
>
> Best regards,
>
> Marcos
>
>
> On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari
> <rahulchaudhari0405@gmail.com <ma...@gmail.com>>
> wrote:
>
> Marcos,
> While configuring hadoop, the "dfs.datanode.data.dir" property
> in hdfs-default.xml should have this list of disks specified on
> separate line. If you specific comma separated list, it will
> replicate on all those disks/partitions.
>
> _Rahul
> Sent from my iPad
>
> > On 13-May-2014, at 12:22 am, Marcos Sousa
> <falecomigo@marcossousa.com <ma...@marcossousa.com>>
> wrote:
> >
> > Hi,
> >
> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use
> them to be my datanode:
> >
> > /vol1/hadoop/data
> > /vol2/hadoop/data
> > /vol3/hadoop/data
> > /volN/hadoop/data
> >
> > How do user those distinct discs not to replicate?
> >
> > Best regards,
> >
> > --
> > Marcos Sousa
>
>
>
>
> --
> Marcos Sousa
> www.marcossousa.com <http://www.marcossousa.com> Enjoy it!
--
*Aitor Pérez*
/Big Data System Engineer/
Telf.: +34 917 680 490
Fax: +34 913 833 301
C/Manuel Tovar, 49-53 - 28034 Madrid - Spain
_http://www.bidoop.es_
Re: Data node with multiple disks
Posted by Marcos Sousa <fa...@marcossousa.com>.
Yes,
I don't want to replicate, just use as one disk? Isn't possible to make
this work?
Best regards,
Marcos
On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari <
rahulchaudhari0405@gmail.com> wrote:
> Marcos,
> While configuring hadoop, the "dfs.datanode.data.dir" property in
> hdfs-default.xml should have this list of disks specified on separate line.
> If you specific comma separated list, it will replicate on all those
> disks/partitions.
>
> _Rahul
> Sent from my iPad
>
> > On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com>
> wrote:
> >
> > Hi,
> >
> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
> my datanode:
> >
> > /vol1/hadoop/data
> > /vol2/hadoop/data
> > /vol3/hadoop/data
> > /volN/hadoop/data
> >
> > How do user those distinct discs not to replicate?
> >
> > Best regards,
> >
> > --
> > Marcos Sousa
>
--
Marcos Sousa
www.marcossousa.com Enjoy it!
Re: Data node with multiple disks
Posted by Marcos Sousa <fa...@marcossousa.com>.
Yes,
I don't want to replicate, just use as one disk? Isn't possible to make
this work?
Best regards,
Marcos
On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari <
rahulchaudhari0405@gmail.com> wrote:
> Marcos,
> While configuring hadoop, the "dfs.datanode.data.dir" property in
> hdfs-default.xml should have this list of disks specified on separate line.
> If you specific comma separated list, it will replicate on all those
> disks/partitions.
>
> _Rahul
> Sent from my iPad
>
> > On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com>
> wrote:
> >
> > Hi,
> >
> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
> my datanode:
> >
> > /vol1/hadoop/data
> > /vol2/hadoop/data
> > /vol3/hadoop/data
> > /volN/hadoop/data
> >
> > How do user those distinct discs not to replicate?
> >
> > Best regards,
> >
> > --
> > Marcos Sousa
>
--
Marcos Sousa
www.marcossousa.com Enjoy it!
Re: Data node with multiple disks
Posted by Marcos Sousa <fa...@marcossousa.com>.
Yes,
I don't want to replicate, just use as one disk? Isn't possible to make
this work?
Best regards,
Marcos
On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari <
rahulchaudhari0405@gmail.com> wrote:
> Marcos,
> While configuring hadoop, the "dfs.datanode.data.dir" property in
> hdfs-default.xml should have this list of disks specified on separate line.
> If you specific comma separated list, it will replicate on all those
> disks/partitions.
>
> _Rahul
> Sent from my iPad
>
> > On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com>
> wrote:
> >
> > Hi,
> >
> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
> my datanode:
> >
> > /vol1/hadoop/data
> > /vol2/hadoop/data
> > /vol3/hadoop/data
> > /volN/hadoop/data
> >
> > How do user those distinct discs not to replicate?
> >
> > Best regards,
> >
> > --
> > Marcos Sousa
>
--
Marcos Sousa
www.marcossousa.com Enjoy it!
Re: Data node with multiple disks
Posted by Marcos Sousa <fa...@marcossousa.com>.
Yes,
I don't want to replicate, just use as one disk? Isn't possible to make
this work?
Best regards,
Marcos
On Tue, May 13, 2014 at 6:55 AM, Rahul Chaudhari <
rahulchaudhari0405@gmail.com> wrote:
> Marcos,
> While configuring hadoop, the "dfs.datanode.data.dir" property in
> hdfs-default.xml should have this list of disks specified on separate line.
> If you specific comma separated list, it will replicate on all those
> disks/partitions.
>
> _Rahul
> Sent from my iPad
>
> > On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com>
> wrote:
> >
> > Hi,
> >
> > I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
> my datanode:
> >
> > /vol1/hadoop/data
> > /vol2/hadoop/data
> > /vol3/hadoop/data
> > /volN/hadoop/data
> >
> > How do user those distinct discs not to replicate?
> >
> > Best regards,
> >
> > --
> > Marcos Sousa
>
--
Marcos Sousa
www.marcossousa.com Enjoy it!
Re: Data node with multiple disks
Posted by Rahul Chaudhari <ra...@gmail.com>.
Marcos,
While configuring hadoop, the "dfs.datanode.data.dir" property in hdfs-default.xml should have this list of disks specified on separate line. If you specific comma separated list, it will replicate on all those disks/partitions.
_Rahul
Sent from my iPad
> On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com> wrote:
>
> Hi,
>
> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be my datanode:
>
> /vol1/hadoop/data
> /vol2/hadoop/data
> /vol3/hadoop/data
> /volN/hadoop/data
>
> How do user those distinct discs not to replicate?
>
> Best regards,
>
> --
> Marcos Sousa
Re: Data node with multiple disks
Posted by Rahul Chaudhari <ra...@gmail.com>.
Marcos,
While configuring hadoop, the "dfs.datanode.data.dir" property in hdfs-default.xml should have this list of disks specified on separate line. If you specific comma separated list, it will replicate on all those disks/partitions.
_Rahul
Sent from my iPad
> On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com> wrote:
>
> Hi,
>
> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be my datanode:
>
> /vol1/hadoop/data
> /vol2/hadoop/data
> /vol3/hadoop/data
> /volN/hadoop/data
>
> How do user those distinct discs not to replicate?
>
> Best regards,
>
> --
> Marcos Sousa
Re: Data node with multiple disks
Posted by Rahul Chaudhari <ra...@gmail.com>.
Marcos,
While configuring hadoop, the "dfs.datanode.data.dir" property in hdfs-default.xml should have this list of disks specified on separate line. If you specific comma separated list, it will replicate on all those disks/partitions.
_Rahul
Sent from my iPad
> On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com> wrote:
>
> Hi,
>
> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be my datanode:
>
> /vol1/hadoop/data
> /vol2/hadoop/data
> /vol3/hadoop/data
> /volN/hadoop/data
>
> How do user those distinct discs not to replicate?
>
> Best regards,
>
> --
> Marcos Sousa
Re: Data node with multiple disks
Posted by Rahul Chaudhari <ra...@gmail.com>.
Marcos,
While configuring hadoop, the "dfs.datanode.data.dir" property in hdfs-default.xml should have this list of disks specified on separate line. If you specific comma separated list, it will replicate on all those disks/partitions.
_Rahul
Sent from my iPad
> On 13-May-2014, at 12:22 am, Marcos Sousa <fa...@marcossousa.com> wrote:
>
> Hi,
>
> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be my datanode:
>
> /vol1/hadoop/data
> /vol2/hadoop/data
> /vol3/hadoop/data
> /volN/hadoop/data
>
> How do user those distinct discs not to replicate?
>
> Best regards,
>
> --
> Marcos Sousa
Re: Data node with multiple disks
Posted by SF Hadoop <sf...@gmail.com>.
Your question is unclear. Please restate and describe what you are
attempting to do.
Thanks.
On Monday, May 12, 2014, Marcos Sousa <fa...@marcossousa.com> wrote:
> Hi,
>
> I have 20 servers with 10 HD with 400GB SATA. I'd like to use them to be
> my datanode:
>
> /vol1/hadoop/data
> /vol2/hadoop/data
> /vol3/hadoop/data
> /volN/hadoop/data
>
> How do user those distinct discs not to replicate?
>
> Best regards,
>
> --
> Marcos Sousa
>