You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Dhanasekaran Anbalagan <bu...@gmail.com> on 2013/02/25 15:15:21 UTC
How to start Data Replicated Blocks in HDFS manually.
Hi Guys,
We have cluster with two data nodes. We configured data replication factor
two.
when i copy data to hdfs, Data's are not fully replicated. It's says * Number
of Under-Replicated Blocks : 15115*
How to manually invoke the Data replication in HDFS.
I restarted cluster also. It's not helps me
Please guide me guys.
-Dhanasekaran.
Did I learn something today? If not, I wasted it.
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Nitin Pawar <ni...@gmail.com>.
as shashwat suggested run the command
$HADOOP_HOME/bin/hadoop dfs -setrep -R -w 2 /
On Mon, Feb 25, 2013 at 8:04 PM, Dhanasekaran Anbalagan
<bu...@gmail.com>wrote:
> HI Nitin
>
> >>did you start the cluster with replication factor 3 and later changed it
> to 2
> yes correct. We have only two datanode, First I wrongly configured after
> I changed to 2.
> and i have done with hadoop fs -setrep -R 2 /myfolder
>
> same thing i done in all folder in HDFS
> >>did you enable rack awareness
> no.
>
> Did I learn something today? If not, I wasted it.
>
>
> On Mon, Feb 25, 2013 at 9:20 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> did you start the cluster with replication factor 3 and later changed it
>> to 2?
>> also did you enable rack awareness in your configs and both the nodes are
>> on same rack?
>>
>>
>>
>>
>> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
>> bugcy013@gmail.com> wrote:
>>
>>> Hi Guys,
>>>
>>> We have cluster with two data nodes. We configured data replication
>>> factor two.
>>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>>> of Under-Replicated Blocks : 15115*
>>> How to manually invoke the Data replication in HDFS.
>>>
>>> I restarted cluster also. It's not helps me
>>>
>>> Please guide me guys.
>>>
>>> -Dhanasekaran.
>>>
>>> Did I learn something today? If not, I wasted it.
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
--
Nitin Pawar
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Nitin Pawar <ni...@gmail.com>.
as shashwat suggested run the command
$HADOOP_HOME/bin/hadoop dfs -setrep -R -w 2 /
On Mon, Feb 25, 2013 at 8:04 PM, Dhanasekaran Anbalagan
<bu...@gmail.com>wrote:
> HI Nitin
>
> >>did you start the cluster with replication factor 3 and later changed it
> to 2
> yes correct. We have only two datanode, First I wrongly configured after
> I changed to 2.
> and i have done with hadoop fs -setrep -R 2 /myfolder
>
> same thing i done in all folder in HDFS
> >>did you enable rack awareness
> no.
>
> Did I learn something today? If not, I wasted it.
>
>
> On Mon, Feb 25, 2013 at 9:20 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> did you start the cluster with replication factor 3 and later changed it
>> to 2?
>> also did you enable rack awareness in your configs and both the nodes are
>> on same rack?
>>
>>
>>
>>
>> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
>> bugcy013@gmail.com> wrote:
>>
>>> Hi Guys,
>>>
>>> We have cluster with two data nodes. We configured data replication
>>> factor two.
>>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>>> of Under-Replicated Blocks : 15115*
>>> How to manually invoke the Data replication in HDFS.
>>>
>>> I restarted cluster also. It's not helps me
>>>
>>> Please guide me guys.
>>>
>>> -Dhanasekaran.
>>>
>>> Did I learn something today? If not, I wasted it.
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
--
Nitin Pawar
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Nitin Pawar <ni...@gmail.com>.
as shashwat suggested run the command
$HADOOP_HOME/bin/hadoop dfs -setrep -R -w 2 /
On Mon, Feb 25, 2013 at 8:04 PM, Dhanasekaran Anbalagan
<bu...@gmail.com>wrote:
> HI Nitin
>
> >>did you start the cluster with replication factor 3 and later changed it
> to 2
> yes correct. We have only two datanode, First I wrongly configured after
> I changed to 2.
> and i have done with hadoop fs -setrep -R 2 /myfolder
>
> same thing i done in all folder in HDFS
> >>did you enable rack awareness
> no.
>
> Did I learn something today? If not, I wasted it.
>
>
> On Mon, Feb 25, 2013 at 9:20 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> did you start the cluster with replication factor 3 and later changed it
>> to 2?
>> also did you enable rack awareness in your configs and both the nodes are
>> on same rack?
>>
>>
>>
>>
>> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
>> bugcy013@gmail.com> wrote:
>>
>>> Hi Guys,
>>>
>>> We have cluster with two data nodes. We configured data replication
>>> factor two.
>>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>>> of Under-Replicated Blocks : 15115*
>>> How to manually invoke the Data replication in HDFS.
>>>
>>> I restarted cluster also. It's not helps me
>>>
>>> Please guide me guys.
>>>
>>> -Dhanasekaran.
>>>
>>> Did I learn something today? If not, I wasted it.
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
--
Nitin Pawar
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Nitin Pawar <ni...@gmail.com>.
as shashwat suggested run the command
$HADOOP_HOME/bin/hadoop dfs -setrep -R -w 2 /
On Mon, Feb 25, 2013 at 8:04 PM, Dhanasekaran Anbalagan
<bu...@gmail.com>wrote:
> HI Nitin
>
> >>did you start the cluster with replication factor 3 and later changed it
> to 2
> yes correct. We have only two datanode, First I wrongly configured after
> I changed to 2.
> and i have done with hadoop fs -setrep -R 2 /myfolder
>
> same thing i done in all folder in HDFS
> >>did you enable rack awareness
> no.
>
> Did I learn something today? If not, I wasted it.
>
>
> On Mon, Feb 25, 2013 at 9:20 AM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> did you start the cluster with replication factor 3 and later changed it
>> to 2?
>> also did you enable rack awareness in your configs and both the nodes are
>> on same rack?
>>
>>
>>
>>
>> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
>> bugcy013@gmail.com> wrote:
>>
>>> Hi Guys,
>>>
>>> We have cluster with two data nodes. We configured data replication
>>> factor two.
>>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>>> of Under-Replicated Blocks : 15115*
>>> How to manually invoke the Data replication in HDFS.
>>>
>>> I restarted cluster also. It's not helps me
>>>
>>> Please guide me guys.
>>>
>>> -Dhanasekaran.
>>>
>>> Did I learn something today? If not, I wasted it.
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
--
Nitin Pawar
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Dhanasekaran Anbalagan <bu...@gmail.com>.
HI Nitin
>>did you start the cluster with replication factor 3 and later changed it
to 2
yes correct. We have only two datanode, First I wrongly configured after I
changed to 2.
and i have done with hadoop fs -setrep -R 2 /myfolder
same thing i done in all folder in HDFS
>>did you enable rack awareness
no.
Did I learn something today? If not, I wasted it.
On Mon, Feb 25, 2013 at 9:20 AM, Nitin Pawar <ni...@gmail.com>wrote:
> did you start the cluster with replication factor 3 and later changed it
> to 2?
> also did you enable rack awareness in your configs and both the nodes are
> on same rack?
>
>
>
>
> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
> bugcy013@gmail.com> wrote:
>
>> Hi Guys,
>>
>> We have cluster with two data nodes. We configured data replication
>> factor two.
>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>> of Under-Replicated Blocks : 15115*
>> How to manually invoke the Data replication in HDFS.
>>
>> I restarted cluster also. It's not helps me
>>
>> Please guide me guys.
>>
>> -Dhanasekaran.
>>
>> Did I learn something today? If not, I wasted it.
>>
>
>
>
> --
> Nitin Pawar
>
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by shashwat shriparv <dw...@gmail.com>.
The problem may be in default replication factor, which is 3, so first
check in hdfs-site.xml for replication factor is specified or not. if it
not add that parameter and restart the cluster ---> first option
2nd Option : change the replicatio factor of the root directory of hdfs to
2 using following command
bin/hadoop dfs -setrep -R -w 2 /
this will chage the replication factor 2 two.
this problem may be also because you have two datanodes and replication
factor is 3. so you can think of the senario when you have two bucket and
you have 3 objects to keep.
∞
Shashwat Shriparv
On Mon, Feb 25, 2013 at 7:50 PM, Nitin Pawar <ni...@gmail.com>wrote:
> did you start the cluster with replication factor 3 and later changed it
> to 2?
> also did you enable rack awareness in your configs and both the nodes are
> on same rack?
>
>
>
>
> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
> bugcy013@gmail.com> wrote:
>
>> Hi Guys,
>>
>> We have cluster with two data nodes. We configured data replication
>> factor two.
>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>> of Under-Replicated Blocks : 15115*
>> How to manually invoke the Data replication in HDFS.
>>
>> I restarted cluster also. It's not helps me
>>
>> Please guide me guys.
>>
>> -Dhanasekaran.
>>
>> Did I learn something today? If not, I wasted it.
>>
>
>
>
> --
> Nitin Pawar
>
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by shashwat shriparv <dw...@gmail.com>.
The problem may be in default replication factor, which is 3, so first
check in hdfs-site.xml for replication factor is specified or not. if it
not add that parameter and restart the cluster ---> first option
2nd Option : change the replicatio factor of the root directory of hdfs to
2 using following command
bin/hadoop dfs -setrep -R -w 2 /
this will chage the replication factor 2 two.
this problem may be also because you have two datanodes and replication
factor is 3. so you can think of the senario when you have two bucket and
you have 3 objects to keep.
∞
Shashwat Shriparv
On Mon, Feb 25, 2013 at 7:50 PM, Nitin Pawar <ni...@gmail.com>wrote:
> did you start the cluster with replication factor 3 and later changed it
> to 2?
> also did you enable rack awareness in your configs and both the nodes are
> on same rack?
>
>
>
>
> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
> bugcy013@gmail.com> wrote:
>
>> Hi Guys,
>>
>> We have cluster with two data nodes. We configured data replication
>> factor two.
>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>> of Under-Replicated Blocks : 15115*
>> How to manually invoke the Data replication in HDFS.
>>
>> I restarted cluster also. It's not helps me
>>
>> Please guide me guys.
>>
>> -Dhanasekaran.
>>
>> Did I learn something today? If not, I wasted it.
>>
>
>
>
> --
> Nitin Pawar
>
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by shashwat shriparv <dw...@gmail.com>.
The problem may be in default replication factor, which is 3, so first
check in hdfs-site.xml for replication factor is specified or not. if it
not add that parameter and restart the cluster ---> first option
2nd Option : change the replicatio factor of the root directory of hdfs to
2 using following command
bin/hadoop dfs -setrep -R -w 2 /
this will chage the replication factor 2 two.
this problem may be also because you have two datanodes and replication
factor is 3. so you can think of the senario when you have two bucket and
you have 3 objects to keep.
∞
Shashwat Shriparv
On Mon, Feb 25, 2013 at 7:50 PM, Nitin Pawar <ni...@gmail.com>wrote:
> did you start the cluster with replication factor 3 and later changed it
> to 2?
> also did you enable rack awareness in your configs and both the nodes are
> on same rack?
>
>
>
>
> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
> bugcy013@gmail.com> wrote:
>
>> Hi Guys,
>>
>> We have cluster with two data nodes. We configured data replication
>> factor two.
>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>> of Under-Replicated Blocks : 15115*
>> How to manually invoke the Data replication in HDFS.
>>
>> I restarted cluster also. It's not helps me
>>
>> Please guide me guys.
>>
>> -Dhanasekaran.
>>
>> Did I learn something today? If not, I wasted it.
>>
>
>
>
> --
> Nitin Pawar
>
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Dhanasekaran Anbalagan <bu...@gmail.com>.
HI Nitin
>>did you start the cluster with replication factor 3 and later changed it
to 2
yes correct. We have only two datanode, First I wrongly configured after I
changed to 2.
and i have done with hadoop fs -setrep -R 2 /myfolder
same thing i done in all folder in HDFS
>>did you enable rack awareness
no.
Did I learn something today? If not, I wasted it.
On Mon, Feb 25, 2013 at 9:20 AM, Nitin Pawar <ni...@gmail.com>wrote:
> did you start the cluster with replication factor 3 and later changed it
> to 2?
> also did you enable rack awareness in your configs and both the nodes are
> on same rack?
>
>
>
>
> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
> bugcy013@gmail.com> wrote:
>
>> Hi Guys,
>>
>> We have cluster with two data nodes. We configured data replication
>> factor two.
>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>> of Under-Replicated Blocks : 15115*
>> How to manually invoke the Data replication in HDFS.
>>
>> I restarted cluster also. It's not helps me
>>
>> Please guide me guys.
>>
>> -Dhanasekaran.
>>
>> Did I learn something today? If not, I wasted it.
>>
>
>
>
> --
> Nitin Pawar
>
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by shashwat shriparv <dw...@gmail.com>.
The problem may be in default replication factor, which is 3, so first
check in hdfs-site.xml for replication factor is specified or not. if it
not add that parameter and restart the cluster ---> first option
2nd Option : change the replicatio factor of the root directory of hdfs to
2 using following command
bin/hadoop dfs -setrep -R -w 2 /
this will chage the replication factor 2 two.
this problem may be also because you have two datanodes and replication
factor is 3. so you can think of the senario when you have two bucket and
you have 3 objects to keep.
∞
Shashwat Shriparv
On Mon, Feb 25, 2013 at 7:50 PM, Nitin Pawar <ni...@gmail.com>wrote:
> did you start the cluster with replication factor 3 and later changed it
> to 2?
> also did you enable rack awareness in your configs and both the nodes are
> on same rack?
>
>
>
>
> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
> bugcy013@gmail.com> wrote:
>
>> Hi Guys,
>>
>> We have cluster with two data nodes. We configured data replication
>> factor two.
>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>> of Under-Replicated Blocks : 15115*
>> How to manually invoke the Data replication in HDFS.
>>
>> I restarted cluster also. It's not helps me
>>
>> Please guide me guys.
>>
>> -Dhanasekaran.
>>
>> Did I learn something today? If not, I wasted it.
>>
>
>
>
> --
> Nitin Pawar
>
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Dhanasekaran Anbalagan <bu...@gmail.com>.
HI Nitin
>>did you start the cluster with replication factor 3 and later changed it
to 2
yes correct. We have only two datanode, First I wrongly configured after I
changed to 2.
and i have done with hadoop fs -setrep -R 2 /myfolder
same thing i done in all folder in HDFS
>>did you enable rack awareness
no.
Did I learn something today? If not, I wasted it.
On Mon, Feb 25, 2013 at 9:20 AM, Nitin Pawar <ni...@gmail.com>wrote:
> did you start the cluster with replication factor 3 and later changed it
> to 2?
> also did you enable rack awareness in your configs and both the nodes are
> on same rack?
>
>
>
>
> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
> bugcy013@gmail.com> wrote:
>
>> Hi Guys,
>>
>> We have cluster with two data nodes. We configured data replication
>> factor two.
>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>> of Under-Replicated Blocks : 15115*
>> How to manually invoke the Data replication in HDFS.
>>
>> I restarted cluster also. It's not helps me
>>
>> Please guide me guys.
>>
>> -Dhanasekaran.
>>
>> Did I learn something today? If not, I wasted it.
>>
>
>
>
> --
> Nitin Pawar
>
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Dhanasekaran Anbalagan <bu...@gmail.com>.
HI Nitin
>>did you start the cluster with replication factor 3 and later changed it
to 2
yes correct. We have only two datanode, First I wrongly configured after I
changed to 2.
and i have done with hadoop fs -setrep -R 2 /myfolder
same thing i done in all folder in HDFS
>>did you enable rack awareness
no.
Did I learn something today? If not, I wasted it.
On Mon, Feb 25, 2013 at 9:20 AM, Nitin Pawar <ni...@gmail.com>wrote:
> did you start the cluster with replication factor 3 and later changed it
> to 2?
> also did you enable rack awareness in your configs and both the nodes are
> on same rack?
>
>
>
>
> On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan <
> bugcy013@gmail.com> wrote:
>
>> Hi Guys,
>>
>> We have cluster with two data nodes. We configured data replication
>> factor two.
>> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
>> of Under-Replicated Blocks : 15115*
>> How to manually invoke the Data replication in HDFS.
>>
>> I restarted cluster also. It's not helps me
>>
>> Please guide me guys.
>>
>> -Dhanasekaran.
>>
>> Did I learn something today? If not, I wasted it.
>>
>
>
>
> --
> Nitin Pawar
>
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Nitin Pawar <ni...@gmail.com>.
did you start the cluster with replication factor 3 and later changed it to
2?
also did you enable rack awareness in your configs and both the nodes are
on same rack?
On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan
<bu...@gmail.com>wrote:
> Hi Guys,
>
> We have cluster with two data nodes. We configured data replication factor
> two.
> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
> of Under-Replicated Blocks : 15115*
> How to manually invoke the Data replication in HDFS.
>
> I restarted cluster also. It's not helps me
>
> Please guide me guys.
>
> -Dhanasekaran.
>
> Did I learn something today? If not, I wasted it.
>
--
Nitin Pawar
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Nitin Pawar <ni...@gmail.com>.
did you start the cluster with replication factor 3 and later changed it to
2?
also did you enable rack awareness in your configs and both the nodes are
on same rack?
On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan
<bu...@gmail.com>wrote:
> Hi Guys,
>
> We have cluster with two data nodes. We configured data replication factor
> two.
> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
> of Under-Replicated Blocks : 15115*
> How to manually invoke the Data replication in HDFS.
>
> I restarted cluster also. It's not helps me
>
> Please guide me guys.
>
> -Dhanasekaran.
>
> Did I learn something today? If not, I wasted it.
>
--
Nitin Pawar
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Nitin Pawar <ni...@gmail.com>.
did you start the cluster with replication factor 3 and later changed it to
2?
also did you enable rack awareness in your configs and both the nodes are
on same rack?
On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan
<bu...@gmail.com>wrote:
> Hi Guys,
>
> We have cluster with two data nodes. We configured data replication factor
> two.
> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
> of Under-Replicated Blocks : 15115*
> How to manually invoke the Data replication in HDFS.
>
> I restarted cluster also. It's not helps me
>
> Please guide me guys.
>
> -Dhanasekaran.
>
> Did I learn something today? If not, I wasted it.
>
--
Nitin Pawar
Re: How to start Data Replicated Blocks in HDFS manually.
Posted by Nitin Pawar <ni...@gmail.com>.
did you start the cluster with replication factor 3 and later changed it to
2?
also did you enable rack awareness in your configs and both the nodes are
on same rack?
On Mon, Feb 25, 2013 at 7:45 PM, Dhanasekaran Anbalagan
<bu...@gmail.com>wrote:
> Hi Guys,
>
> We have cluster with two data nodes. We configured data replication factor
> two.
> when i copy data to hdfs, Data's are not fully replicated. It's says * Number
> of Under-Replicated Blocks : 15115*
> How to manually invoke the Data replication in HDFS.
>
> I restarted cluster also. It's not helps me
>
> Please guide me guys.
>
> -Dhanasekaran.
>
> Did I learn something today? If not, I wasted it.
>
--
Nitin Pawar