You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Joarder KAMAL <jo...@gmail.com> on 2013/05/03 08:32:33 UTC
How can I add a new hard disk in an existing HDFS cluster?
Hi,
I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
drive /dev/vdb of 60GB size and want to add it into my existing HDFS
cluster. How can I format the new hard disk (and in which format? XFS?) and
mount it to work with HDFS
Default HDFS directory is situated in
/usr/local/hadoop-1.0.4/hadoop-datastore
And I followed this link for installation.
http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
Many thanks in advance :)
Regards,
Joarder Kamal
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Ling Kun <lk...@gmail.com>.
hi,Joarder Kamal:
The following steps may help.
1. Stop your datanode daemon which you want to add a new HDD.
2. modify the data.dfs.dir property in hdfs-site.xml as mentioned by
Geelong Yao.
3. start the DataNode daemon again.
Ling Kun
On Mon, May 6, 2013 at 12:30 AM, Rahul Bhattacharjee <
rahul.rec.dgp@gmail.com> wrote:
> I think the question here is as how to add new HDD volumn into an already
> existing formatted HDFS cluster.
> Not sure , by just adding the directory in data.dfs.dir would help.
>
>
> On Fri, May 3, 2013 at 3:28 PM, Håvard Wahl Kongsgård <
> haavard.kongsgaard@gmail.com> wrote:
>
>> go for ext3 or ext4
>>
>>
>> On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and
>>> the initial hard disk (/dev/vda1) size is 10G only. Now I have a second
>>> hard drive /dev/vdb of 60GB size and want to add it into my existing HDFS
>>> cluster. How can I format the new hard disk (and in which format? XFS?) and
>>> mount it to work with HDFS
>>>
>>> Default HDFS directory is situated in
>>> /usr/local/hadoop-1.0.4/hadoop-datastore
>>> And I followed this link for installation.
>>>
>>> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>>>
>>> Many thanks in advance :)
>>>
>>>
>>> Regards,
>>> Joarder Kamal
>>>
>>
>>
>>
>> --
>> Håvard Wahl Kongsgård
>> Data Scientist
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.dbkeeping.com/
>>
>
>
--
http://www.lingcc.com
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Ling Kun <lk...@gmail.com>.
hi,Joarder Kamal:
The following steps may help.
1. Stop your datanode daemon which you want to add a new HDD.
2. modify the data.dfs.dir property in hdfs-site.xml as mentioned by
Geelong Yao.
3. start the DataNode daemon again.
Ling Kun
On Mon, May 6, 2013 at 12:30 AM, Rahul Bhattacharjee <
rahul.rec.dgp@gmail.com> wrote:
> I think the question here is as how to add new HDD volumn into an already
> existing formatted HDFS cluster.
> Not sure , by just adding the directory in data.dfs.dir would help.
>
>
> On Fri, May 3, 2013 at 3:28 PM, Håvard Wahl Kongsgård <
> haavard.kongsgaard@gmail.com> wrote:
>
>> go for ext3 or ext4
>>
>>
>> On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and
>>> the initial hard disk (/dev/vda1) size is 10G only. Now I have a second
>>> hard drive /dev/vdb of 60GB size and want to add it into my existing HDFS
>>> cluster. How can I format the new hard disk (and in which format? XFS?) and
>>> mount it to work with HDFS
>>>
>>> Default HDFS directory is situated in
>>> /usr/local/hadoop-1.0.4/hadoop-datastore
>>> And I followed this link for installation.
>>>
>>> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>>>
>>> Many thanks in advance :)
>>>
>>>
>>> Regards,
>>> Joarder Kamal
>>>
>>
>>
>>
>> --
>> Håvard Wahl Kongsgård
>> Data Scientist
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.dbkeeping.com/
>>
>
>
--
http://www.lingcc.com
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Ling Kun <lk...@gmail.com>.
hi,Joarder Kamal:
The following steps may help.
1. Stop your datanode daemon which you want to add a new HDD.
2. modify the data.dfs.dir property in hdfs-site.xml as mentioned by
Geelong Yao.
3. start the DataNode daemon again.
Ling Kun
On Mon, May 6, 2013 at 12:30 AM, Rahul Bhattacharjee <
rahul.rec.dgp@gmail.com> wrote:
> I think the question here is as how to add new HDD volumn into an already
> existing formatted HDFS cluster.
> Not sure , by just adding the directory in data.dfs.dir would help.
>
>
> On Fri, May 3, 2013 at 3:28 PM, Håvard Wahl Kongsgård <
> haavard.kongsgaard@gmail.com> wrote:
>
>> go for ext3 or ext4
>>
>>
>> On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and
>>> the initial hard disk (/dev/vda1) size is 10G only. Now I have a second
>>> hard drive /dev/vdb of 60GB size and want to add it into my existing HDFS
>>> cluster. How can I format the new hard disk (and in which format? XFS?) and
>>> mount it to work with HDFS
>>>
>>> Default HDFS directory is situated in
>>> /usr/local/hadoop-1.0.4/hadoop-datastore
>>> And I followed this link for installation.
>>>
>>> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>>>
>>> Many thanks in advance :)
>>>
>>>
>>> Regards,
>>> Joarder Kamal
>>>
>>
>>
>>
>> --
>> Håvard Wahl Kongsgård
>> Data Scientist
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.dbkeeping.com/
>>
>
>
--
http://www.lingcc.com
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Ling Kun <lk...@gmail.com>.
hi,Joarder Kamal:
The following steps may help.
1. Stop your datanode daemon which you want to add a new HDD.
2. modify the data.dfs.dir property in hdfs-site.xml as mentioned by
Geelong Yao.
3. start the DataNode daemon again.
Ling Kun
On Mon, May 6, 2013 at 12:30 AM, Rahul Bhattacharjee <
rahul.rec.dgp@gmail.com> wrote:
> I think the question here is as how to add new HDD volumn into an already
> existing formatted HDFS cluster.
> Not sure , by just adding the directory in data.dfs.dir would help.
>
>
> On Fri, May 3, 2013 at 3:28 PM, Håvard Wahl Kongsgård <
> haavard.kongsgaard@gmail.com> wrote:
>
>> go for ext3 or ext4
>>
>>
>> On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and
>>> the initial hard disk (/dev/vda1) size is 10G only. Now I have a second
>>> hard drive /dev/vdb of 60GB size and want to add it into my existing HDFS
>>> cluster. How can I format the new hard disk (and in which format? XFS?) and
>>> mount it to work with HDFS
>>>
>>> Default HDFS directory is situated in
>>> /usr/local/hadoop-1.0.4/hadoop-datastore
>>> And I followed this link for installation.
>>>
>>> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>>>
>>> Many thanks in advance :)
>>>
>>>
>>> Regards,
>>> Joarder Kamal
>>>
>>
>>
>>
>> --
>> Håvard Wahl Kongsgård
>> Data Scientist
>> Faculty of Medicine &
>> Department of Mathematical Sciences
>> NTNU
>>
>> http://havard.dbkeeping.com/
>>
>
>
--
http://www.lingcc.com
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I think the question here is as how to add new HDD volumn into an already
existing formatted HDFS cluster.
Not sure , by just adding the directory in data.dfs.dir would help.
On Fri, May 3, 2013 at 3:28 PM, Håvard Wahl Kongsgård <
haavard.kongsgaard@gmail.com> wrote:
> go for ext3 or ext4
>
>
> On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
>
>> Hi,
>>
>> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
>> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
>> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
>> cluster. How can I format the new hard disk (and in which format? XFS?) and
>> mount it to work with HDFS
>>
>> Default HDFS directory is situated in
>> /usr/local/hadoop-1.0.4/hadoop-datastore
>> And I followed this link for installation.
>>
>> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>>
>> Many thanks in advance :)
>>
>>
>> Regards,
>> Joarder Kamal
>>
>
>
>
> --
> Håvard Wahl Kongsgård
> Data Scientist
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.dbkeeping.com/
>
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I think the question here is as how to add new HDD volumn into an already
existing formatted HDFS cluster.
Not sure , by just adding the directory in data.dfs.dir would help.
On Fri, May 3, 2013 at 3:28 PM, Håvard Wahl Kongsgård <
haavard.kongsgaard@gmail.com> wrote:
> go for ext3 or ext4
>
>
> On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
>
>> Hi,
>>
>> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
>> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
>> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
>> cluster. How can I format the new hard disk (and in which format? XFS?) and
>> mount it to work with HDFS
>>
>> Default HDFS directory is situated in
>> /usr/local/hadoop-1.0.4/hadoop-datastore
>> And I followed this link for installation.
>>
>> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>>
>> Many thanks in advance :)
>>
>>
>> Regards,
>> Joarder Kamal
>>
>
>
>
> --
> Håvard Wahl Kongsgård
> Data Scientist
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.dbkeeping.com/
>
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I think the question here is as how to add new HDD volumn into an already
existing formatted HDFS cluster.
Not sure , by just adding the directory in data.dfs.dir would help.
On Fri, May 3, 2013 at 3:28 PM, Håvard Wahl Kongsgård <
haavard.kongsgaard@gmail.com> wrote:
> go for ext3 or ext4
>
>
> On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
>
>> Hi,
>>
>> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
>> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
>> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
>> cluster. How can I format the new hard disk (and in which format? XFS?) and
>> mount it to work with HDFS
>>
>> Default HDFS directory is situated in
>> /usr/local/hadoop-1.0.4/hadoop-datastore
>> And I followed this link for installation.
>>
>> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>>
>> Many thanks in advance :)
>>
>>
>> Regards,
>> Joarder Kamal
>>
>
>
>
> --
> Håvard Wahl Kongsgård
> Data Scientist
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.dbkeeping.com/
>
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Rahul Bhattacharjee <ra...@gmail.com>.
I think the question here is as how to add new HDD volumn into an already
existing formatted HDFS cluster.
Not sure , by just adding the directory in data.dfs.dir would help.
On Fri, May 3, 2013 at 3:28 PM, Håvard Wahl Kongsgård <
haavard.kongsgaard@gmail.com> wrote:
> go for ext3 or ext4
>
>
> On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
>
>> Hi,
>>
>> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
>> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
>> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
>> cluster. How can I format the new hard disk (and in which format? XFS?) and
>> mount it to work with HDFS
>>
>> Default HDFS directory is situated in
>> /usr/local/hadoop-1.0.4/hadoop-datastore
>> And I followed this link for installation.
>>
>> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>>
>> Many thanks in advance :)
>>
>>
>> Regards,
>> Joarder Kamal
>>
>
>
>
> --
> Håvard Wahl Kongsgård
> Data Scientist
> Faculty of Medicine &
> Department of Mathematical Sciences
> NTNU
>
> http://havard.dbkeeping.com/
>
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
go for ext3 or ext4
On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
> Hi,
>
> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
> cluster. How can I format the new hard disk (and in which format? XFS?) and
> mount it to work with HDFS
>
> Default HDFS directory is situated in
> /usr/local/hadoop-1.0.4/hadoop-datastore
> And I followed this link for installation.
>
> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>
> Many thanks in advance :)
>
>
> Regards,
> Joarder Kamal
>
--
Håvard Wahl Kongsgård
Data Scientist
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.dbkeeping.com/
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
go for ext3 or ext4
On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
> Hi,
>
> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
> cluster. How can I format the new hard disk (and in which format? XFS?) and
> mount it to work with HDFS
>
> Default HDFS directory is situated in
> /usr/local/hadoop-1.0.4/hadoop-datastore
> And I followed this link for installation.
>
> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>
> Many thanks in advance :)
>
>
> Regards,
> Joarder Kamal
>
--
Håvard Wahl Kongsgård
Data Scientist
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.dbkeeping.com/
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
go for ext3 or ext4
On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
> Hi,
>
> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
> cluster. How can I format the new hard disk (and in which format? XFS?) and
> mount it to work with HDFS
>
> Default HDFS directory is situated in
> /usr/local/hadoop-1.0.4/hadoop-datastore
> And I followed this link for installation.
>
> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>
> Many thanks in advance :)
>
>
> Regards,
> Joarder Kamal
>
--
Håvard Wahl Kongsgård
Data Scientist
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.dbkeeping.com/
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Geelong Yao <ge...@gmail.com>.
you can change the setting of data.dfs.dir in hdfs-site.xml if your version
is 1.x
<property>
<name>data.dfs.dir</name>
<value>/usr/hadoop/tmp/dfs/data, /dev/vdb </value>
</property>
2013/5/3 Joarder KAMAL <jo...@gmail.com>
> Hi,
>
> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
> cluster. How can I format the new hard disk (and in which format? XFS?) and
> mount it to work with HDFS
>
> Default HDFS directory is situated in
> /usr/local/hadoop-1.0.4/hadoop-datastore
> And I followed this link for installation.
>
> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>
> Many thanks in advance :)
>
>
> Regards,
> Joarder Kamal
>
--
>From Good To Great
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Geelong Yao <ge...@gmail.com>.
you can change the setting of data.dfs.dir in hdfs-site.xml if your version
is 1.x
<property>
<name>data.dfs.dir</name>
<value>/usr/hadoop/tmp/dfs/data, /dev/vdb </value>
</property>
2013/5/3 Joarder KAMAL <jo...@gmail.com>
> Hi,
>
> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
> cluster. How can I format the new hard disk (and in which format? XFS?) and
> mount it to work with HDFS
>
> Default HDFS directory is situated in
> /usr/local/hadoop-1.0.4/hadoop-datastore
> And I followed this link for installation.
>
> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>
> Many thanks in advance :)
>
>
> Regards,
> Joarder Kamal
>
--
>From Good To Great
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Geelong Yao <ge...@gmail.com>.
you can change the setting of data.dfs.dir in hdfs-site.xml if your version
is 1.x
<property>
<name>data.dfs.dir</name>
<value>/usr/hadoop/tmp/dfs/data, /dev/vdb </value>
</property>
2013/5/3 Joarder KAMAL <jo...@gmail.com>
> Hi,
>
> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
> cluster. How can I format the new hard disk (and in which format? XFS?) and
> mount it to work with HDFS
>
> Default HDFS directory is situated in
> /usr/local/hadoop-1.0.4/hadoop-datastore
> And I followed this link for installation.
>
> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>
> Many thanks in advance :)
>
>
> Regards,
> Joarder Kamal
>
--
>From Good To Great
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Håvard Wahl Kongsgård <ha...@gmail.com>.
go for ext3 or ext4
On Fri, May 3, 2013 at 8:32 AM, Joarder KAMAL <jo...@gmail.com> wrote:
> Hi,
>
> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
> cluster. How can I format the new hard disk (and in which format? XFS?) and
> mount it to work with HDFS
>
> Default HDFS directory is situated in
> /usr/local/hadoop-1.0.4/hadoop-datastore
> And I followed this link for installation.
>
> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>
> Many thanks in advance :)
>
>
> Regards,
> Joarder Kamal
>
--
Håvard Wahl Kongsgård
Data Scientist
Faculty of Medicine &
Department of Mathematical Sciences
NTNU
http://havard.dbkeeping.com/
Re: How can I add a new hard disk in an existing HDFS cluster?
Posted by Geelong Yao <ge...@gmail.com>.
you can change the setting of data.dfs.dir in hdfs-site.xml if your version
is 1.x
<property>
<name>data.dfs.dir</name>
<value>/usr/hadoop/tmp/dfs/data, /dev/vdb </value>
</property>
2013/5/3 Joarder KAMAL <jo...@gmail.com>
> Hi,
>
> I have a running HDFS cluster (Hadoop/HBase) consists of 4 nodes and the
> initial hard disk (/dev/vda1) size is 10G only. Now I have a second hard
> drive /dev/vdb of 60GB size and want to add it into my existing HDFS
> cluster. How can I format the new hard disk (and in which format? XFS?) and
> mount it to work with HDFS
>
> Default HDFS directory is situated in
> /usr/local/hadoop-1.0.4/hadoop-datastore
> And I followed this link for installation.
>
> http://ankitasblogger.blogspot.com.au/2011/01/hadoop-cluster-setup.html
>
> Many thanks in advance :)
>
>
> Regards,
> Joarder Kamal
>
--
>From Good To Great