You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by "Kilaru, Sambaiah" <Sa...@intuit.com> on 2014/06/26 15:02:13 UTC
How to make hdfs data rack aware
Hi All,
We setup a cluster with out rackawareness, now that we got right rackawreness script, we wanted to
Make sure data is rack aware.
How we make sure existing data is rack aware and NN should move copies if all of them are in the
Same rack?
Thanks,
Sam
Re: How to make hdfs data rack aware
Posted by Adam Kawa <ka...@gmail.com>.
You can run
$ sudo -u hdfs hdfs dfsadmin -report | grep "Hostname" -A 1
2014-07-02 7:33 GMT+02:00 hadoop hive <ha...@gmail.com>:
> Try running fsck, it will also validate the block placement as well as
> replication.
> On Jun 27, 2014 6:49 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
> wrote:
>
>> My topology script is working fine for data I am writing to hdfs. My
>> question is how to make the
>> Existing data topoloy compaliant?
>>
>> Thanks,
>> sam
>>
>> From: karthikeyan S <ka...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Thursday, June 26, 2014 at 8:23 PM
>> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Subject: Re: How to make hdfs data rack aware
>>
>> Hi
>>
>> Did you try supplying the script name in hdfs-site.xml as the value for
>> the parameter topology.script.file.name?
>>
>> Thanks
>>
>> Karthik
>> On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> We setup a cluster with out rackawareness, now that we got right
>>> rackawreness script, we wanted to
>>> Make sure data is rack aware.
>>> How we make sure existing data is rack aware and NN should move copies
>>> if all of them are in the
>>> Same rack?
>>>
>>> Thanks,
>>> Sam
>>>
>>
Re: How to make hdfs data rack aware
Posted by Adam Kawa <ka...@gmail.com>.
You can run
$ sudo -u hdfs hdfs dfsadmin -report | grep "Hostname" -A 1
2014-07-02 7:33 GMT+02:00 hadoop hive <ha...@gmail.com>:
> Try running fsck, it will also validate the block placement as well as
> replication.
> On Jun 27, 2014 6:49 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
> wrote:
>
>> My topology script is working fine for data I am writing to hdfs. My
>> question is how to make the
>> Existing data topoloy compaliant?
>>
>> Thanks,
>> sam
>>
>> From: karthikeyan S <ka...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Thursday, June 26, 2014 at 8:23 PM
>> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Subject: Re: How to make hdfs data rack aware
>>
>> Hi
>>
>> Did you try supplying the script name in hdfs-site.xml as the value for
>> the parameter topology.script.file.name?
>>
>> Thanks
>>
>> Karthik
>> On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> We setup a cluster with out rackawareness, now that we got right
>>> rackawreness script, we wanted to
>>> Make sure data is rack aware.
>>> How we make sure existing data is rack aware and NN should move copies
>>> if all of them are in the
>>> Same rack?
>>>
>>> Thanks,
>>> Sam
>>>
>>
Re: How to make hdfs data rack aware
Posted by Adam Kawa <ka...@gmail.com>.
You can run
$ sudo -u hdfs hdfs dfsadmin -report | grep "Hostname" -A 1
2014-07-02 7:33 GMT+02:00 hadoop hive <ha...@gmail.com>:
> Try running fsck, it will also validate the block placement as well as
> replication.
> On Jun 27, 2014 6:49 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
> wrote:
>
>> My topology script is working fine for data I am writing to hdfs. My
>> question is how to make the
>> Existing data topoloy compaliant?
>>
>> Thanks,
>> sam
>>
>> From: karthikeyan S <ka...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Thursday, June 26, 2014 at 8:23 PM
>> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Subject: Re: How to make hdfs data rack aware
>>
>> Hi
>>
>> Did you try supplying the script name in hdfs-site.xml as the value for
>> the parameter topology.script.file.name?
>>
>> Thanks
>>
>> Karthik
>> On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> We setup a cluster with out rackawareness, now that we got right
>>> rackawreness script, we wanted to
>>> Make sure data is rack aware.
>>> How we make sure existing data is rack aware and NN should move copies
>>> if all of them are in the
>>> Same rack?
>>>
>>> Thanks,
>>> Sam
>>>
>>
Re: How to make hdfs data rack aware
Posted by Adam Kawa <ka...@gmail.com>.
You can run
$ sudo -u hdfs hdfs dfsadmin -report | grep "Hostname" -A 1
2014-07-02 7:33 GMT+02:00 hadoop hive <ha...@gmail.com>:
> Try running fsck, it will also validate the block placement as well as
> replication.
> On Jun 27, 2014 6:49 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
> wrote:
>
>> My topology script is working fine for data I am writing to hdfs. My
>> question is how to make the
>> Existing data topoloy compaliant?
>>
>> Thanks,
>> sam
>>
>> From: karthikeyan S <ka...@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Date: Thursday, June 26, 2014 at 8:23 PM
>> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
>> Subject: Re: How to make hdfs data rack aware
>>
>> Hi
>>
>> Did you try supplying the script name in hdfs-site.xml as the value for
>> the parameter topology.script.file.name?
>>
>> Thanks
>>
>> Karthik
>> On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> We setup a cluster with out rackawareness, now that we got right
>>> rackawreness script, we wanted to
>>> Make sure data is rack aware.
>>> How we make sure existing data is rack aware and NN should move copies
>>> if all of them are in the
>>> Same rack?
>>>
>>> Thanks,
>>> Sam
>>>
>>
Re: How to make hdfs data rack aware
Posted by hadoop hive <ha...@gmail.com>.
Try running fsck, it will also validate the block placement as well as
replication.
On Jun 27, 2014 6:49 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
wrote:
> My topology script is working fine for data I am writing to hdfs. My
> question is how to make the
> Existing data topoloy compaliant?
>
> Thanks,
> sam
>
> From: karthikeyan S <ka...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday, June 26, 2014 at 8:23 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: How to make hdfs data rack aware
>
> Hi
>
> Did you try supplying the script name in hdfs-site.xml as the value for
> the parameter topology.script.file.name?
>
> Thanks
>
> Karthik
> On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
> wrote:
>
>> Hi All,
>>
>> We setup a cluster with out rackawareness, now that we got right
>> rackawreness script, we wanted to
>> Make sure data is rack aware.
>> How we make sure existing data is rack aware and NN should move copies if
>> all of them are in the
>> Same rack?
>>
>> Thanks,
>> Sam
>>
>
Re: How to make hdfs data rack aware
Posted by hadoop hive <ha...@gmail.com>.
Try running fsck, it will also validate the block placement as well as
replication.
On Jun 27, 2014 6:49 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
wrote:
> My topology script is working fine for data I am writing to hdfs. My
> question is how to make the
> Existing data topoloy compaliant?
>
> Thanks,
> sam
>
> From: karthikeyan S <ka...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday, June 26, 2014 at 8:23 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: How to make hdfs data rack aware
>
> Hi
>
> Did you try supplying the script name in hdfs-site.xml as the value for
> the parameter topology.script.file.name?
>
> Thanks
>
> Karthik
> On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
> wrote:
>
>> Hi All,
>>
>> We setup a cluster with out rackawareness, now that we got right
>> rackawreness script, we wanted to
>> Make sure data is rack aware.
>> How we make sure existing data is rack aware and NN should move copies if
>> all of them are in the
>> Same rack?
>>
>> Thanks,
>> Sam
>>
>
Re: How to make hdfs data rack aware
Posted by hadoop hive <ha...@gmail.com>.
Try running fsck, it will also validate the block placement as well as
replication.
On Jun 27, 2014 6:49 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
wrote:
> My topology script is working fine for data I am writing to hdfs. My
> question is how to make the
> Existing data topoloy compaliant?
>
> Thanks,
> sam
>
> From: karthikeyan S <ka...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday, June 26, 2014 at 8:23 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: How to make hdfs data rack aware
>
> Hi
>
> Did you try supplying the script name in hdfs-site.xml as the value for
> the parameter topology.script.file.name?
>
> Thanks
>
> Karthik
> On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
> wrote:
>
>> Hi All,
>>
>> We setup a cluster with out rackawareness, now that we got right
>> rackawreness script, we wanted to
>> Make sure data is rack aware.
>> How we make sure existing data is rack aware and NN should move copies if
>> all of them are in the
>> Same rack?
>>
>> Thanks,
>> Sam
>>
>
Re: How to make hdfs data rack aware
Posted by hadoop hive <ha...@gmail.com>.
Try running fsck, it will also validate the block placement as well as
replication.
On Jun 27, 2014 6:49 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
wrote:
> My topology script is working fine for data I am writing to hdfs. My
> question is how to make the
> Existing data topoloy compaliant?
>
> Thanks,
> sam
>
> From: karthikeyan S <ka...@gmail.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Thursday, June 26, 2014 at 8:23 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: How to make hdfs data rack aware
>
> Hi
>
> Did you try supplying the script name in hdfs-site.xml as the value for
> the parameter topology.script.file.name?
>
> Thanks
>
> Karthik
> On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
> wrote:
>
>> Hi All,
>>
>> We setup a cluster with out rackawareness, now that we got right
>> rackawreness script, we wanted to
>> Make sure data is rack aware.
>> How we make sure existing data is rack aware and NN should move copies if
>> all of them are in the
>> Same rack?
>>
>> Thanks,
>> Sam
>>
>
Re: How to make hdfs data rack aware
Posted by "Kilaru, Sambaiah" <Sa...@intuit.com>.
My topology script is working fine for data I am writing to hdfs. My question is how to make the
Existing data topoloy compaliant?
Thanks,
sam
From: karthikeyan S <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, June 26, 2014 at 8:23 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: How to make hdfs data rack aware
Hi
Did you try supplying the script name in hdfs-site.xml as the value for the parameter topology.script.file.name<http://topology.script.file.name>?
Thanks
Karthik
On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>> wrote:
Hi All,
We setup a cluster with out rackawareness, now that we got right rackawreness script, we wanted to
Make sure data is rack aware.
How we make sure existing data is rack aware and NN should move copies if all of them are in the
Same rack?
Thanks,
Sam
Re: How to make hdfs data rack aware
Posted by "Kilaru, Sambaiah" <Sa...@intuit.com>.
My topology script is working fine for data I am writing to hdfs. My question is how to make the
Existing data topoloy compaliant?
Thanks,
sam
From: karthikeyan S <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, June 26, 2014 at 8:23 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: How to make hdfs data rack aware
Hi
Did you try supplying the script name in hdfs-site.xml as the value for the parameter topology.script.file.name<http://topology.script.file.name>?
Thanks
Karthik
On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>> wrote:
Hi All,
We setup a cluster with out rackawareness, now that we got right rackawreness script, we wanted to
Make sure data is rack aware.
How we make sure existing data is rack aware and NN should move copies if all of them are in the
Same rack?
Thanks,
Sam
Re: How to make hdfs data rack aware
Posted by "Kilaru, Sambaiah" <Sa...@intuit.com>.
My topology script is working fine for data I am writing to hdfs. My question is how to make the
Existing data topoloy compaliant?
Thanks,
sam
From: karthikeyan S <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, June 26, 2014 at 8:23 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: How to make hdfs data rack aware
Hi
Did you try supplying the script name in hdfs-site.xml as the value for the parameter topology.script.file.name<http://topology.script.file.name>?
Thanks
Karthik
On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>> wrote:
Hi All,
We setup a cluster with out rackawareness, now that we got right rackawreness script, we wanted to
Make sure data is rack aware.
How we make sure existing data is rack aware and NN should move copies if all of them are in the
Same rack?
Thanks,
Sam
Re: How to make hdfs data rack aware
Posted by "Kilaru, Sambaiah" <Sa...@intuit.com>.
My topology script is working fine for data I am writing to hdfs. My question is how to make the
Existing data topoloy compaliant?
Thanks,
sam
From: karthikeyan S <ka...@gmail.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Thursday, June 26, 2014 at 8:23 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: How to make hdfs data rack aware
Hi
Did you try supplying the script name in hdfs-site.xml as the value for the parameter topology.script.file.name<http://topology.script.file.name>?
Thanks
Karthik
On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>> wrote:
Hi All,
We setup a cluster with out rackawareness, now that we got right rackawreness script, we wanted to
Make sure data is rack aware.
How we make sure existing data is rack aware and NN should move copies if all of them are in the
Same rack?
Thanks,
Sam
Re: How to make hdfs data rack aware
Posted by karthikeyan S <ka...@gmail.com>.
Hi
Did you try supplying the script name in hdfs-site.xml as the value for the
parameter topology.script.file.name?
Thanks
Karthik
On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
wrote:
> Hi All,
>
> We setup a cluster with out rackawareness, now that we got right
> rackawreness script, we wanted to
> Make sure data is rack aware.
> How we make sure existing data is rack aware and NN should move copies if
> all of them are in the
> Same rack?
>
> Thanks,
> Sam
>
Re: How to make hdfs data rack aware
Posted by karthikeyan S <ka...@gmail.com>.
Hi
Did you try supplying the script name in hdfs-site.xml as the value for the
parameter topology.script.file.name?
Thanks
Karthik
On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
wrote:
> Hi All,
>
> We setup a cluster with out rackawareness, now that we got right
> rackawreness script, we wanted to
> Make sure data is rack aware.
> How we make sure existing data is rack aware and NN should move copies if
> all of them are in the
> Same rack?
>
> Thanks,
> Sam
>
Re: How to make hdfs data rack aware
Posted by karthikeyan S <ka...@gmail.com>.
Hi
Did you try supplying the script name in hdfs-site.xml as the value for the
parameter topology.script.file.name?
Thanks
Karthik
On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
wrote:
> Hi All,
>
> We setup a cluster with out rackawareness, now that we got right
> rackawreness script, we wanted to
> Make sure data is rack aware.
> How we make sure existing data is rack aware and NN should move copies if
> all of them are in the
> Same rack?
>
> Thanks,
> Sam
>
Re: How to make hdfs data rack aware
Posted by karthikeyan S <ka...@gmail.com>.
Hi
Did you try supplying the script name in hdfs-site.xml as the value for the
parameter topology.script.file.name?
Thanks
Karthik
On Jun 26, 2014 6:02 AM, "Kilaru, Sambaiah" <Sa...@intuit.com>
wrote:
> Hi All,
>
> We setup a cluster with out rackawareness, now that we got right
> rackawreness script, we wanted to
> Make sure data is rack aware.
> How we make sure existing data is rack aware and NN should move copies if
> all of them are in the
> Same rack?
>
> Thanks,
> Sam
>