You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Francis.Hu" <fr...@reachjunction.com> on 2013/07/02 05:30:12 UTC
答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
Re: 答复: 答复: a question about dfs.replication
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Hu
It comes an point in my mind.And I tested it.
Before you set the dfs.replication to 2,the block may have already
existed with the original replication value 3.
After you changed the value, the replication of the former block was
still 3.
So the file that you created after the changement will be with the
replication 2.
However I do not have the access to change the replication within
the existed
files.
2013/7/2 Francis.Hu <fr...@reachjunction.com>
> Yes , it returns 2 correctly after "hdfs getconf -confkey
> dfs.replication"****
>
> ** **
>
> ****
>
> ** **
>
> but in web page ,it is 3 as below:****
>
> ****
>
> ** **
>
> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
> *����ʱ��:* Monday, July 01, 2013 23:24
> *�ռ���:* user@hadoop.apache.org
> *����:* Re: ��: a question about dfs.replication****
>
> ** **
>
> Hi ****
>
> ** **
>
> Could you please get the property value by using : hdfs getconf
> -confkey dfs.replication.****
>
>
> 发自我的 iPhone****
>
>
> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>
> ****
>
> Actually, My java client is running with the same configuration as the
> hadoop's . The dfs.replication is already set as 2 in my hadoop's
> configuration.****
>
> So i think the dfs.replication is already overrided by my configuration in
> hdfs-site.xml. but seems it doesn't work even i overrided the parameter
> evidently.****
>
> ****
>
> ****
>
> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>
> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
> *收件�?span lang="EN-US">:* user@hadoop.apache.org
> *主�**��:* Re: a question about dfs.replication****
>
> ****
>
> On 01.07.2013 10:19, Francis.Hu wrote:****
>
> Hi, All****
>
> ****
>
> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and
> two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After
> all configuration work is done, I started all nodes. Then I saved a file
> into HDFS through java client. nOW I can access hdfs web page:
> x.x.x.x:50070,and also see the file is already listed in the hdfs list.***
> *
>
> My question is: *The replication column in HDFS web page is showing as
> 3, not 2. Does anyone know What the problem is?*****
>
> * *****
>
> ---Actual setting of hdfs-site.xml****
>
> <property>****
>
> <name>dfs.replication</name>****
>
> <value>2</value>****
>
> </property>****
>
> ****
>
> After that, I typed dfsamdin command to check the file:****
>
> hdfs fsck /test3/****
>
> The result of above command:****
>
> /test3/hello005.txt: Under replicated
> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
> Target Replicas is 3 but found 2 replica(s).****
>
> Status: HEALTHY****
>
> Total size: 35 B****
>
> Total dirs: 1****
>
> Total files: 1****
>
> Total blocks (validated): 1 (avg. block size 35 B)****
>
> Minimally replicated blocks: 1 (100.0 %)****
>
> Over-replicated blocks: 0 (0.0 %)****
>
> Under-replicated blocks: 1 (100.0 %)****
>
> Mis-replicated blocks: 0 (0.0 %)****
>
> Default replication factor: 2****
>
> Average block replication: 2.0****
>
> Corrupt blocks: 0****
>
> Missing replicas: 1 (33.333332 %)****
>
> Number of data-nodes: 3****
>
> Number of racks: 1****
>
> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>
> ****
>
> ****
>
> Thanks,****
>
> Francis Hu****
>
> ****
>
> If I'm not mistaking "dfs.replication" parameter in config sets only
> default replication factor, which can be overrided when putting file to
> hdfs.****
>
>
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
YouPeng Yang,
you said that may be the answer. Thank you.
发件人: YouPeng Yang [mailto:yypvsxf19870706@gmail.com]
发送时间: Tuesday, July 02, 2013 12:52
收件人: user@hadoop.apache.org
主题: Re: reply: a question about dfs.replication
HI HU and Yu
Aggree with dfs.replication is a client side configuration, not server side. It make the point in my last mail sense.
And the cmd:hdfs dfs -setrep -R -w 2 / solve the problem that I can not change the existed file's replication value.
2013/7/2 Azuryy Yu <az...@gmail.com>
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you need to set it to '2' on your client side( your application running on). THEN execute command such as : hdfs dfs -put or call HDFS API in java application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com> wrote:
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
YouPeng Yang,
you said that may be the answer. Thank you.
发件人: YouPeng Yang [mailto:yypvsxf19870706@gmail.com]
发送时间: Tuesday, July 02, 2013 12:52
收件人: user@hadoop.apache.org
主题: Re: reply: a question about dfs.replication
HI HU and Yu
Aggree with dfs.replication is a client side configuration, not server side. It make the point in my last mail sense.
And the cmd:hdfs dfs -setrep -R -w 2 / solve the problem that I can not change the existed file's replication value.
2013/7/2 Azuryy Yu <az...@gmail.com>
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you need to set it to '2' on your client side( your application running on). THEN execute command such as : hdfs dfs -put or call HDFS API in java application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com> wrote:
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
YouPeng Yang,
you said that may be the answer. Thank you.
发件人: YouPeng Yang [mailto:yypvsxf19870706@gmail.com]
发送时间: Tuesday, July 02, 2013 12:52
收件人: user@hadoop.apache.org
主题: Re: reply: a question about dfs.replication
HI HU and Yu
Aggree with dfs.replication is a client side configuration, not server side. It make the point in my last mail sense.
And the cmd:hdfs dfs -setrep -R -w 2 / solve the problem that I can not change the existed file's replication value.
2013/7/2 Azuryy Yu <az...@gmail.com>
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you need to set it to '2' on your client side( your application running on). THEN execute command such as : hdfs dfs -put or call HDFS API in java application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com> wrote:
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
YouPeng Yang,
you said that may be the answer. Thank you.
发件人: YouPeng Yang [mailto:yypvsxf19870706@gmail.com]
发送时间: Tuesday, July 02, 2013 12:52
收件人: user@hadoop.apache.org
主题: Re: reply: a question about dfs.replication
HI HU and Yu
Aggree with dfs.replication is a client side configuration, not server side. It make the point in my last mail sense.
And the cmd:hdfs dfs -setrep -R -w 2 / solve the problem that I can not change the existed file's replication value.
2013/7/2 Azuryy Yu <az...@gmail.com>
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you need to set it to '2' on your client side( your application running on). THEN execute command such as : hdfs dfs -put or call HDFS API in java application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com> wrote:
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
Re: reply: a question about dfs.replication
Posted by YouPeng Yang <yy...@gmail.com>.
HI HU and Yu
Aggree with dfs.replication is a client side configuration, not server
side. It make the point in my last mail sense.
And the cmd:hdfs dfs -setrep -R -w 2 / solve the problem that I can
not change the existed file's replication value.
2013/7/2 Azuryy Yu <az...@gmail.com>
> It's not HDFS issue.
>
> dfs.replication is a client side configuration, not server side. so you
> need to set it to '2' on your client side( your application running on).
> THEN execute command such as : hdfs dfs -put or call HDFS API in java
> application.
>
>
>
>
> On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com>wrote:
>
>> Thanks all of you, I just get the problem fixed through the command: ***
>> *
>>
>> hdfs dfs -setrep -R -w 2 /****
>>
>> ** **
>>
>> Is that an issue of HDFS ? Why do i need to execute manually a command to
>> tell the hadoop the replication factor even it is set in hdfs-site.xml ?*
>> ***
>>
>> ** **
>>
>> Thanks,****
>>
>> Francis.Hu****
>>
>> ** **
>>
>> *������:* Francis.Hu [mailto:francis.hu@reachjunction.com]
>> *����ʱ��:* Tuesday, July 02, 2013 11:30
>> *�ռ���:* user@hadoop.apache.org
>> *����:* ��: ��: a question about dfs.replication****
>>
>> ** **
>>
>> Yes , it returns 2 correctly after "hdfs getconf -confkey
>> dfs.replication"****
>>
>> ** **
>>
>> ****
>>
>> ** **
>>
>> but in web page ,it is 3 as below:****
>>
>> ****
>>
>> ** **
>>
>> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
>> *����ʱ��:* Monday, July 01, 2013 23:24
>> *�ռ���:* user@hadoop.apache.org
>> *����:* Re: ��: a question about dfs.replication****
>>
>> ** **
>>
>> Hi ****
>>
>> ** **
>>
>> Could you please get the property value by using : hdfs getconf
>> -confkey dfs.replication.****
>>
>>
>> 发自我的 iPhone****
>>
>>
>> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>>
>> ****
>>
>> Actually, My java client is running with the same configuration as the
>> hadoop's . The dfs.replication is already set as 2 in my hadoop's
>> configuration.****
>>
>> So i think the dfs.replication is already overrided by my configuration
>> in hdfs-site.xml. but seems it doesn't work even i overrided the parameter
>> evidently.****
>>
>> ****
>>
>> ****
>>
>> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>>
>> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
>> *收件�?span lang="EN-US">:* user@hadoop.apache.org
>> *主�**��:* Re: a question about dfs.replication****
>>
>> ****
>>
>> On 01.07.2013 10:19, Francis.Hu wrote:****
>>
>> Hi, All****
>>
>> ****
>>
>> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode
>> and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml.
>> After all configuration work is done, I started all nodes. Then I saved a
>> file into HDFS through java client. nOW I can access hdfs web page:
>> x.x.x.x:50070,and also see the file is already listed in the hdfs list.**
>> **
>>
>> My question is: *The replication column in HDFS web page is showing as
>> 3, not 2. Does anyone know What the problem is?*****
>>
>> * *****
>>
>> ---Actual setting of hdfs-site.xml****
>>
>> <property>****
>>
>> <name>dfs.replication</name>****
>>
>> <value>2</value>****
>>
>> </property>****
>>
>> ****
>>
>> After that, I typed dfsamdin command to check the file:****
>>
>> hdfs fsck /test3/****
>>
>> The result of above command:****
>>
>> /test3/hello005.txt: Under replicated
>> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
>> Target Replicas is 3 but found 2 replica(s).****
>>
>> Status: HEALTHY****
>>
>> Total size: 35 B****
>>
>> Total dirs: 1****
>>
>> Total files: 1****
>>
>> Total blocks (validated): 1 (avg. block size 35 B)****
>>
>> Minimally replicated blocks: 1 (100.0 %)****
>>
>> Over-replicated blocks: 0 (0.0 %)****
>>
>> Under-replicated blocks: 1 (100.0 %)****
>>
>> Mis-replicated blocks: 0 (0.0 %)****
>>
>> Default replication factor: 2****
>>
>> Average block replication: 2.0****
>>
>> Corrupt blocks: 0****
>>
>> Missing replicas: 1 (33.333332 %)****
>>
>> Number of data-nodes: 3****
>>
>> Number of racks: 1****
>>
>> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>>
>> ****
>>
>> ****
>>
>> Thanks,****
>>
>> Francis Hu****
>>
>> ****
>>
>> If I'm not mistaking "dfs.replication" parameter in config sets only
>> default replication factor, which can be overrided when putting file to
>> hdfs.****
>>
>>
>
Re: reply: a question about dfs.replication
Posted by YouPeng Yang <yy...@gmail.com>.
HI HU and Yu
Aggree with dfs.replication is a client side configuration, not server
side. It make the point in my last mail sense.
And the cmd:hdfs dfs -setrep -R -w 2 / solve the problem that I can
not change the existed file's replication value.
2013/7/2 Azuryy Yu <az...@gmail.com>
> It's not HDFS issue.
>
> dfs.replication is a client side configuration, not server side. so you
> need to set it to '2' on your client side( your application running on).
> THEN execute command such as : hdfs dfs -put or call HDFS API in java
> application.
>
>
>
>
> On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com>wrote:
>
>> Thanks all of you, I just get the problem fixed through the command: ***
>> *
>>
>> hdfs dfs -setrep -R -w 2 /****
>>
>> ** **
>>
>> Is that an issue of HDFS ? Why do i need to execute manually a command to
>> tell the hadoop the replication factor even it is set in hdfs-site.xml ?*
>> ***
>>
>> ** **
>>
>> Thanks,****
>>
>> Francis.Hu****
>>
>> ** **
>>
>> *������:* Francis.Hu [mailto:francis.hu@reachjunction.com]
>> *����ʱ��:* Tuesday, July 02, 2013 11:30
>> *�ռ���:* user@hadoop.apache.org
>> *����:* ��: ��: a question about dfs.replication****
>>
>> ** **
>>
>> Yes , it returns 2 correctly after "hdfs getconf -confkey
>> dfs.replication"****
>>
>> ** **
>>
>> ****
>>
>> ** **
>>
>> but in web page ,it is 3 as below:****
>>
>> ****
>>
>> ** **
>>
>> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
>> *����ʱ��:* Monday, July 01, 2013 23:24
>> *�ռ���:* user@hadoop.apache.org
>> *����:* Re: ��: a question about dfs.replication****
>>
>> ** **
>>
>> Hi ****
>>
>> ** **
>>
>> Could you please get the property value by using : hdfs getconf
>> -confkey dfs.replication.****
>>
>>
>> 发自我的 iPhone****
>>
>>
>> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>>
>> ****
>>
>> Actually, My java client is running with the same configuration as the
>> hadoop's . The dfs.replication is already set as 2 in my hadoop's
>> configuration.****
>>
>> So i think the dfs.replication is already overrided by my configuration
>> in hdfs-site.xml. but seems it doesn't work even i overrided the parameter
>> evidently.****
>>
>> ****
>>
>> ****
>>
>> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>>
>> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
>> *收件�?span lang="EN-US">:* user@hadoop.apache.org
>> *主�**��:* Re: a question about dfs.replication****
>>
>> ****
>>
>> On 01.07.2013 10:19, Francis.Hu wrote:****
>>
>> Hi, All****
>>
>> ****
>>
>> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode
>> and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml.
>> After all configuration work is done, I started all nodes. Then I saved a
>> file into HDFS through java client. nOW I can access hdfs web page:
>> x.x.x.x:50070,and also see the file is already listed in the hdfs list.**
>> **
>>
>> My question is: *The replication column in HDFS web page is showing as
>> 3, not 2. Does anyone know What the problem is?*****
>>
>> * *****
>>
>> ---Actual setting of hdfs-site.xml****
>>
>> <property>****
>>
>> <name>dfs.replication</name>****
>>
>> <value>2</value>****
>>
>> </property>****
>>
>> ****
>>
>> After that, I typed dfsamdin command to check the file:****
>>
>> hdfs fsck /test3/****
>>
>> The result of above command:****
>>
>> /test3/hello005.txt: Under replicated
>> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
>> Target Replicas is 3 but found 2 replica(s).****
>>
>> Status: HEALTHY****
>>
>> Total size: 35 B****
>>
>> Total dirs: 1****
>>
>> Total files: 1****
>>
>> Total blocks (validated): 1 (avg. block size 35 B)****
>>
>> Minimally replicated blocks: 1 (100.0 %)****
>>
>> Over-replicated blocks: 0 (0.0 %)****
>>
>> Under-replicated blocks: 1 (100.0 %)****
>>
>> Mis-replicated blocks: 0 (0.0 %)****
>>
>> Default replication factor: 2****
>>
>> Average block replication: 2.0****
>>
>> Corrupt blocks: 0****
>>
>> Missing replicas: 1 (33.333332 %)****
>>
>> Number of data-nodes: 3****
>>
>> Number of racks: 1****
>>
>> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>>
>> ****
>>
>> ****
>>
>> Thanks,****
>>
>> Francis Hu****
>>
>> ****
>>
>> If I'm not mistaking "dfs.replication" parameter in config sets only
>> default replication factor, which can be overrided when putting file to
>> hdfs.****
>>
>>
>
Re: reply: a question about dfs.replication
Posted by YouPeng Yang <yy...@gmail.com>.
HI HU and Yu
Aggree with dfs.replication is a client side configuration, not server
side. It make the point in my last mail sense.
And the cmd:hdfs dfs -setrep -R -w 2 / solve the problem that I can
not change the existed file's replication value.
2013/7/2 Azuryy Yu <az...@gmail.com>
> It's not HDFS issue.
>
> dfs.replication is a client side configuration, not server side. so you
> need to set it to '2' on your client side( your application running on).
> THEN execute command such as : hdfs dfs -put or call HDFS API in java
> application.
>
>
>
>
> On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com>wrote:
>
>> Thanks all of you, I just get the problem fixed through the command: ***
>> *
>>
>> hdfs dfs -setrep -R -w 2 /****
>>
>> ** **
>>
>> Is that an issue of HDFS ? Why do i need to execute manually a command to
>> tell the hadoop the replication factor even it is set in hdfs-site.xml ?*
>> ***
>>
>> ** **
>>
>> Thanks,****
>>
>> Francis.Hu****
>>
>> ** **
>>
>> *������:* Francis.Hu [mailto:francis.hu@reachjunction.com]
>> *����ʱ��:* Tuesday, July 02, 2013 11:30
>> *�ռ���:* user@hadoop.apache.org
>> *����:* ��: ��: a question about dfs.replication****
>>
>> ** **
>>
>> Yes , it returns 2 correctly after "hdfs getconf -confkey
>> dfs.replication"****
>>
>> ** **
>>
>> ****
>>
>> ** **
>>
>> but in web page ,it is 3 as below:****
>>
>> ****
>>
>> ** **
>>
>> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
>> *����ʱ��:* Monday, July 01, 2013 23:24
>> *�ռ���:* user@hadoop.apache.org
>> *����:* Re: ��: a question about dfs.replication****
>>
>> ** **
>>
>> Hi ****
>>
>> ** **
>>
>> Could you please get the property value by using : hdfs getconf
>> -confkey dfs.replication.****
>>
>>
>> 发自我的 iPhone****
>>
>>
>> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>>
>> ****
>>
>> Actually, My java client is running with the same configuration as the
>> hadoop's . The dfs.replication is already set as 2 in my hadoop's
>> configuration.****
>>
>> So i think the dfs.replication is already overrided by my configuration
>> in hdfs-site.xml. but seems it doesn't work even i overrided the parameter
>> evidently.****
>>
>> ****
>>
>> ****
>>
>> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>>
>> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
>> *收件�?span lang="EN-US">:* user@hadoop.apache.org
>> *主�**��:* Re: a question about dfs.replication****
>>
>> ****
>>
>> On 01.07.2013 10:19, Francis.Hu wrote:****
>>
>> Hi, All****
>>
>> ****
>>
>> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode
>> and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml.
>> After all configuration work is done, I started all nodes. Then I saved a
>> file into HDFS through java client. nOW I can access hdfs web page:
>> x.x.x.x:50070,and also see the file is already listed in the hdfs list.**
>> **
>>
>> My question is: *The replication column in HDFS web page is showing as
>> 3, not 2. Does anyone know What the problem is?*****
>>
>> * *****
>>
>> ---Actual setting of hdfs-site.xml****
>>
>> <property>****
>>
>> <name>dfs.replication</name>****
>>
>> <value>2</value>****
>>
>> </property>****
>>
>> ****
>>
>> After that, I typed dfsamdin command to check the file:****
>>
>> hdfs fsck /test3/****
>>
>> The result of above command:****
>>
>> /test3/hello005.txt: Under replicated
>> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
>> Target Replicas is 3 but found 2 replica(s).****
>>
>> Status: HEALTHY****
>>
>> Total size: 35 B****
>>
>> Total dirs: 1****
>>
>> Total files: 1****
>>
>> Total blocks (validated): 1 (avg. block size 35 B)****
>>
>> Minimally replicated blocks: 1 (100.0 %)****
>>
>> Over-replicated blocks: 0 (0.0 %)****
>>
>> Under-replicated blocks: 1 (100.0 %)****
>>
>> Mis-replicated blocks: 0 (0.0 %)****
>>
>> Default replication factor: 2****
>>
>> Average block replication: 2.0****
>>
>> Corrupt blocks: 0****
>>
>> Missing replicas: 1 (33.333332 %)****
>>
>> Number of data-nodes: 3****
>>
>> Number of racks: 1****
>>
>> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>>
>> ****
>>
>> ****
>>
>> Thanks,****
>>
>> Francis Hu****
>>
>> ****
>>
>> If I'm not mistaking "dfs.replication" parameter in config sets only
>> default replication factor, which can be overrided when putting file to
>> hdfs.****
>>
>>
>
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
actually, my client side is already set to "2".
发件人: Azuryy Yu [mailto:azuryyyu@gmail.com]
发送时间: Tuesday, July 02, 2013 12:40
收件人: user@hadoop.apache.org
主题: Re: reply: a question about dfs.replication
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you need to set it to '2' on your client side( your application running on). THEN execute command such as : hdfs dfs -put or call HDFS API in java application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com> wrote:
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
actually, my client side is already set to "2".
发件人: Azuryy Yu [mailto:azuryyyu@gmail.com]
发送时间: Tuesday, July 02, 2013 12:40
收件人: user@hadoop.apache.org
主题: Re: reply: a question about dfs.replication
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you need to set it to '2' on your client side( your application running on). THEN execute command such as : hdfs dfs -put or call HDFS API in java application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com> wrote:
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
actually, my client side is already set to "2".
发件人: Azuryy Yu [mailto:azuryyyu@gmail.com]
发送时间: Tuesday, July 02, 2013 12:40
收件人: user@hadoop.apache.org
主题: Re: reply: a question about dfs.replication
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you need to set it to '2' on your client side( your application running on). THEN execute command such as : hdfs dfs -put or call HDFS API in java application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com> wrote:
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
Re: reply: a question about dfs.replication
Posted by YouPeng Yang <yy...@gmail.com>.
HI HU and Yu
Aggree with dfs.replication is a client side configuration, not server
side. It make the point in my last mail sense.
And the cmd:hdfs dfs -setrep -R -w 2 / solve the problem that I can
not change the existed file's replication value.
2013/7/2 Azuryy Yu <az...@gmail.com>
> It's not HDFS issue.
>
> dfs.replication is a client side configuration, not server side. so you
> need to set it to '2' on your client side( your application running on).
> THEN execute command such as : hdfs dfs -put or call HDFS API in java
> application.
>
>
>
>
> On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com>wrote:
>
>> Thanks all of you, I just get the problem fixed through the command: ***
>> *
>>
>> hdfs dfs -setrep -R -w 2 /****
>>
>> ** **
>>
>> Is that an issue of HDFS ? Why do i need to execute manually a command to
>> tell the hadoop the replication factor even it is set in hdfs-site.xml ?*
>> ***
>>
>> ** **
>>
>> Thanks,****
>>
>> Francis.Hu****
>>
>> ** **
>>
>> *������:* Francis.Hu [mailto:francis.hu@reachjunction.com]
>> *����ʱ��:* Tuesday, July 02, 2013 11:30
>> *�ռ���:* user@hadoop.apache.org
>> *����:* ��: ��: a question about dfs.replication****
>>
>> ** **
>>
>> Yes , it returns 2 correctly after "hdfs getconf -confkey
>> dfs.replication"****
>>
>> ** **
>>
>> ****
>>
>> ** **
>>
>> but in web page ,it is 3 as below:****
>>
>> ****
>>
>> ** **
>>
>> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
>> *����ʱ��:* Monday, July 01, 2013 23:24
>> *�ռ���:* user@hadoop.apache.org
>> *����:* Re: ��: a question about dfs.replication****
>>
>> ** **
>>
>> Hi ****
>>
>> ** **
>>
>> Could you please get the property value by using : hdfs getconf
>> -confkey dfs.replication.****
>>
>>
>> 发自我的 iPhone****
>>
>>
>> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>>
>> ****
>>
>> Actually, My java client is running with the same configuration as the
>> hadoop's . The dfs.replication is already set as 2 in my hadoop's
>> configuration.****
>>
>> So i think the dfs.replication is already overrided by my configuration
>> in hdfs-site.xml. but seems it doesn't work even i overrided the parameter
>> evidently.****
>>
>> ****
>>
>> ****
>>
>> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>>
>> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
>> *收件�?span lang="EN-US">:* user@hadoop.apache.org
>> *主�**��:* Re: a question about dfs.replication****
>>
>> ****
>>
>> On 01.07.2013 10:19, Francis.Hu wrote:****
>>
>> Hi, All****
>>
>> ****
>>
>> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode
>> and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml.
>> After all configuration work is done, I started all nodes. Then I saved a
>> file into HDFS through java client. nOW I can access hdfs web page:
>> x.x.x.x:50070,and also see the file is already listed in the hdfs list.**
>> **
>>
>> My question is: *The replication column in HDFS web page is showing as
>> 3, not 2. Does anyone know What the problem is?*****
>>
>> * *****
>>
>> ---Actual setting of hdfs-site.xml****
>>
>> <property>****
>>
>> <name>dfs.replication</name>****
>>
>> <value>2</value>****
>>
>> </property>****
>>
>> ****
>>
>> After that, I typed dfsamdin command to check the file:****
>>
>> hdfs fsck /test3/****
>>
>> The result of above command:****
>>
>> /test3/hello005.txt: Under replicated
>> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
>> Target Replicas is 3 but found 2 replica(s).****
>>
>> Status: HEALTHY****
>>
>> Total size: 35 B****
>>
>> Total dirs: 1****
>>
>> Total files: 1****
>>
>> Total blocks (validated): 1 (avg. block size 35 B)****
>>
>> Minimally replicated blocks: 1 (100.0 %)****
>>
>> Over-replicated blocks: 0 (0.0 %)****
>>
>> Under-replicated blocks: 1 (100.0 %)****
>>
>> Mis-replicated blocks: 0 (0.0 %)****
>>
>> Default replication factor: 2****
>>
>> Average block replication: 2.0****
>>
>> Corrupt blocks: 0****
>>
>> Missing replicas: 1 (33.333332 %)****
>>
>> Number of data-nodes: 3****
>>
>> Number of racks: 1****
>>
>> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>>
>> ****
>>
>> ****
>>
>> Thanks,****
>>
>> Francis Hu****
>>
>> ****
>>
>> If I'm not mistaking "dfs.replication" parameter in config sets only
>> default replication factor, which can be overrided when putting file to
>> hdfs.****
>>
>>
>
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
actually, my client side is already set to "2".
发件人: Azuryy Yu [mailto:azuryyyu@gmail.com]
发送时间: Tuesday, July 02, 2013 12:40
收件人: user@hadoop.apache.org
主题: Re: reply: a question about dfs.replication
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you need to set it to '2' on your client side( your application running on). THEN execute command such as : hdfs dfs -put or call HDFS API in java application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com> wrote:
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
Re: reply: a question about dfs.replication
Posted by Azuryy Yu <az...@gmail.com>.
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you
need to set it to '2' on your client side( your application running on).
THEN execute command such as : hdfs dfs -put or call HDFS API in java
application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Thanks all of you, I just get the problem fixed through the command: ****
>
> hdfs dfs -setrep -R -w 2 /****
>
> ** **
>
> Is that an issue of HDFS ? Why do i need to execute manually a command to
> tell the hadoop the replication factor even it is set in hdfs-site.xml ?**
> **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
> *������:* Francis.Hu [mailto:francis.hu@reachjunction.com]
> *����ʱ��:* Tuesday, July 02, 2013 11:30
> *�ռ���:* user@hadoop.apache.org
> *����:* ��: ��: a question about dfs.replication****
>
> ** **
>
> Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
> ****
>
> ** **
>
> ****
>
> ** **
>
> but in web page ,it is 3 as below:****
>
> ****
>
> ** **
>
> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
> *����ʱ��:* Monday, July 01, 2013 23:24
> *�ռ���:* user@hadoop.apache.org
> *����:* Re: ��: a question about dfs.replication****
>
> ** **
>
> Hi ****
>
> ** **
>
> Could you please get the property value by using : hdfs getconf
> -confkey dfs.replication.****
>
>
> 发自我的 iPhone****
>
>
> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>
> ****
>
> Actually, My java client is running with the same configuration as the
> hadoop's . The dfs.replication is already set as 2 in my hadoop's
> configuration.****
>
> So i think the dfs.replication is already overrided by my configuration in
> hdfs-site.xml. but seems it doesn't work even i overrided the parameter
> evidently.****
>
> ****
>
> ****
>
> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>
> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
> *收件�?span lang="EN-US">:* user@hadoop.apache.org
> *主�**��:* Re: a question about dfs.replication****
>
> ****
>
> On 01.07.2013 10:19, Francis.Hu wrote:****
>
> Hi, All****
>
> ****
>
> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and
> two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After
> all configuration work is done, I started all nodes. Then I saved a file
> into HDFS through java client. nOW I can access hdfs web page:
> x.x.x.x:50070,and also see the file is already listed in the hdfs list.***
> *
>
> My question is: *The replication column in HDFS web page is showing as
> 3, not 2. Does anyone know What the problem is?*****
>
> * *****
>
> ---Actual setting of hdfs-site.xml****
>
> <property>****
>
> <name>dfs.replication</name>****
>
> <value>2</value>****
>
> </property>****
>
> ****
>
> After that, I typed dfsamdin command to check the file:****
>
> hdfs fsck /test3/****
>
> The result of above command:****
>
> /test3/hello005.txt: Under replicated
> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
> Target Replicas is 3 but found 2 replica(s).****
>
> Status: HEALTHY****
>
> Total size: 35 B****
>
> Total dirs: 1****
>
> Total files: 1****
>
> Total blocks (validated): 1 (avg. block size 35 B)****
>
> Minimally replicated blocks: 1 (100.0 %)****
>
> Over-replicated blocks: 0 (0.0 %)****
>
> Under-replicated blocks: 1 (100.0 %)****
>
> Mis-replicated blocks: 0 (0.0 %)****
>
> Default replication factor: 2****
>
> Average block replication: 2.0****
>
> Corrupt blocks: 0****
>
> Missing replicas: 1 (33.333332 %)****
>
> Number of data-nodes: 3****
>
> Number of racks: 1****
>
> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>
> ****
>
> ****
>
> Thanks,****
>
> Francis Hu****
>
> ****
>
> If I'm not mistaking "dfs.replication" parameter in config sets only
> default replication factor, which can be overrided when putting file to
> hdfs.****
>
>
Re: reply: a question about dfs.replication
Posted by Azuryy Yu <az...@gmail.com>.
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you
need to set it to '2' on your client side( your application running on).
THEN execute command such as : hdfs dfs -put or call HDFS API in java
application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Thanks all of you, I just get the problem fixed through the command: ****
>
> hdfs dfs -setrep -R -w 2 /****
>
> ** **
>
> Is that an issue of HDFS ? Why do i need to execute manually a command to
> tell the hadoop the replication factor even it is set in hdfs-site.xml ?**
> **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
> *������:* Francis.Hu [mailto:francis.hu@reachjunction.com]
> *����ʱ��:* Tuesday, July 02, 2013 11:30
> *�ռ���:* user@hadoop.apache.org
> *����:* ��: ��: a question about dfs.replication****
>
> ** **
>
> Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
> ****
>
> ** **
>
> ****
>
> ** **
>
> but in web page ,it is 3 as below:****
>
> ****
>
> ** **
>
> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
> *����ʱ��:* Monday, July 01, 2013 23:24
> *�ռ���:* user@hadoop.apache.org
> *����:* Re: ��: a question about dfs.replication****
>
> ** **
>
> Hi ****
>
> ** **
>
> Could you please get the property value by using : hdfs getconf
> -confkey dfs.replication.****
>
>
> 发自我的 iPhone****
>
>
> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>
> ****
>
> Actually, My java client is running with the same configuration as the
> hadoop's . The dfs.replication is already set as 2 in my hadoop's
> configuration.****
>
> So i think the dfs.replication is already overrided by my configuration in
> hdfs-site.xml. but seems it doesn't work even i overrided the parameter
> evidently.****
>
> ****
>
> ****
>
> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>
> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
> *收件�?span lang="EN-US">:* user@hadoop.apache.org
> *主�**��:* Re: a question about dfs.replication****
>
> ****
>
> On 01.07.2013 10:19, Francis.Hu wrote:****
>
> Hi, All****
>
> ****
>
> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and
> two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After
> all configuration work is done, I started all nodes. Then I saved a file
> into HDFS through java client. nOW I can access hdfs web page:
> x.x.x.x:50070,and also see the file is already listed in the hdfs list.***
> *
>
> My question is: *The replication column in HDFS web page is showing as
> 3, not 2. Does anyone know What the problem is?*****
>
> * *****
>
> ---Actual setting of hdfs-site.xml****
>
> <property>****
>
> <name>dfs.replication</name>****
>
> <value>2</value>****
>
> </property>****
>
> ****
>
> After that, I typed dfsamdin command to check the file:****
>
> hdfs fsck /test3/****
>
> The result of above command:****
>
> /test3/hello005.txt: Under replicated
> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
> Target Replicas is 3 but found 2 replica(s).****
>
> Status: HEALTHY****
>
> Total size: 35 B****
>
> Total dirs: 1****
>
> Total files: 1****
>
> Total blocks (validated): 1 (avg. block size 35 B)****
>
> Minimally replicated blocks: 1 (100.0 %)****
>
> Over-replicated blocks: 0 (0.0 %)****
>
> Under-replicated blocks: 1 (100.0 %)****
>
> Mis-replicated blocks: 0 (0.0 %)****
>
> Default replication factor: 2****
>
> Average block replication: 2.0****
>
> Corrupt blocks: 0****
>
> Missing replicas: 1 (33.333332 %)****
>
> Number of data-nodes: 3****
>
> Number of racks: 1****
>
> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>
> ****
>
> ****
>
> Thanks,****
>
> Francis Hu****
>
> ****
>
> If I'm not mistaking "dfs.replication" parameter in config sets only
> default replication factor, which can be overrided when putting file to
> hdfs.****
>
>
Re: reply: a question about dfs.replication
Posted by Azuryy Yu <az...@gmail.com>.
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you
need to set it to '2' on your client side( your application running on).
THEN execute command such as : hdfs dfs -put or call HDFS API in java
application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Thanks all of you, I just get the problem fixed through the command: ****
>
> hdfs dfs -setrep -R -w 2 /****
>
> ** **
>
> Is that an issue of HDFS ? Why do i need to execute manually a command to
> tell the hadoop the replication factor even it is set in hdfs-site.xml ?**
> **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
> *������:* Francis.Hu [mailto:francis.hu@reachjunction.com]
> *����ʱ��:* Tuesday, July 02, 2013 11:30
> *�ռ���:* user@hadoop.apache.org
> *����:* ��: ��: a question about dfs.replication****
>
> ** **
>
> Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
> ****
>
> ** **
>
> ****
>
> ** **
>
> but in web page ,it is 3 as below:****
>
> ****
>
> ** **
>
> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
> *����ʱ��:* Monday, July 01, 2013 23:24
> *�ռ���:* user@hadoop.apache.org
> *����:* Re: ��: a question about dfs.replication****
>
> ** **
>
> Hi ****
>
> ** **
>
> Could you please get the property value by using : hdfs getconf
> -confkey dfs.replication.****
>
>
> 发自我的 iPhone****
>
>
> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>
> ****
>
> Actually, My java client is running with the same configuration as the
> hadoop's . The dfs.replication is already set as 2 in my hadoop's
> configuration.****
>
> So i think the dfs.replication is already overrided by my configuration in
> hdfs-site.xml. but seems it doesn't work even i overrided the parameter
> evidently.****
>
> ****
>
> ****
>
> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>
> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
> *收件�?span lang="EN-US">:* user@hadoop.apache.org
> *主�**��:* Re: a question about dfs.replication****
>
> ****
>
> On 01.07.2013 10:19, Francis.Hu wrote:****
>
> Hi, All****
>
> ****
>
> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and
> two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After
> all configuration work is done, I started all nodes. Then I saved a file
> into HDFS through java client. nOW I can access hdfs web page:
> x.x.x.x:50070,and also see the file is already listed in the hdfs list.***
> *
>
> My question is: *The replication column in HDFS web page is showing as
> 3, not 2. Does anyone know What the problem is?*****
>
> * *****
>
> ---Actual setting of hdfs-site.xml****
>
> <property>****
>
> <name>dfs.replication</name>****
>
> <value>2</value>****
>
> </property>****
>
> ****
>
> After that, I typed dfsamdin command to check the file:****
>
> hdfs fsck /test3/****
>
> The result of above command:****
>
> /test3/hello005.txt: Under replicated
> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
> Target Replicas is 3 but found 2 replica(s).****
>
> Status: HEALTHY****
>
> Total size: 35 B****
>
> Total dirs: 1****
>
> Total files: 1****
>
> Total blocks (validated): 1 (avg. block size 35 B)****
>
> Minimally replicated blocks: 1 (100.0 %)****
>
> Over-replicated blocks: 0 (0.0 %)****
>
> Under-replicated blocks: 1 (100.0 %)****
>
> Mis-replicated blocks: 0 (0.0 %)****
>
> Default replication factor: 2****
>
> Average block replication: 2.0****
>
> Corrupt blocks: 0****
>
> Missing replicas: 1 (33.333332 %)****
>
> Number of data-nodes: 3****
>
> Number of racks: 1****
>
> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>
> ****
>
> ****
>
> Thanks,****
>
> Francis Hu****
>
> ****
>
> If I'm not mistaking "dfs.replication" parameter in config sets only
> default replication factor, which can be overrided when putting file to
> hdfs.****
>
>
Re: reply: a question about dfs.replication
Posted by Azuryy Yu <az...@gmail.com>.
It's not HDFS issue.
dfs.replication is a client side configuration, not server side. so you
need to set it to '2' on your client side( your application running on).
THEN execute command such as : hdfs dfs -put or call HDFS API in java
application.
On Tue, Jul 2, 2013 at 12:25 PM, Francis.Hu <fr...@reachjunction.com>wrote:
> Thanks all of you, I just get the problem fixed through the command: ****
>
> hdfs dfs -setrep -R -w 2 /****
>
> ** **
>
> Is that an issue of HDFS ? Why do i need to execute manually a command to
> tell the hadoop the replication factor even it is set in hdfs-site.xml ?**
> **
>
> ** **
>
> Thanks,****
>
> Francis.Hu****
>
> ** **
>
> *������:* Francis.Hu [mailto:francis.hu@reachjunction.com]
> *����ʱ��:* Tuesday, July 02, 2013 11:30
> *�ռ���:* user@hadoop.apache.org
> *����:* ��: ��: a question about dfs.replication****
>
> ** **
>
> Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
> ****
>
> ** **
>
> ****
>
> ** **
>
> but in web page ,it is 3 as below:****
>
> ****
>
> ** **
>
> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
> *����ʱ��:* Monday, July 01, 2013 23:24
> *�ռ���:* user@hadoop.apache.org
> *����:* Re: ��: a question about dfs.replication****
>
> ** **
>
> Hi ****
>
> ** **
>
> Could you please get the property value by using : hdfs getconf
> -confkey dfs.replication.****
>
>
> 发自我的 iPhone****
>
>
> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>
> ****
>
> Actually, My java client is running with the same configuration as the
> hadoop's . The dfs.replication is already set as 2 in my hadoop's
> configuration.****
>
> So i think the dfs.replication is already overrided by my configuration in
> hdfs-site.xml. but seems it doesn't work even i overrided the parameter
> evidently.****
>
> ****
>
> ****
>
> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>
> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
> *收件�?span lang="EN-US">:* user@hadoop.apache.org
> *主�**��:* Re: a question about dfs.replication****
>
> ****
>
> On 01.07.2013 10:19, Francis.Hu wrote:****
>
> Hi, All****
>
> ****
>
> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and
> two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After
> all configuration work is done, I started all nodes. Then I saved a file
> into HDFS through java client. nOW I can access hdfs web page:
> x.x.x.x:50070,and also see the file is already listed in the hdfs list.***
> *
>
> My question is: *The replication column in HDFS web page is showing as
> 3, not 2. Does anyone know What the problem is?*****
>
> * *****
>
> ---Actual setting of hdfs-site.xml****
>
> <property>****
>
> <name>dfs.replication</name>****
>
> <value>2</value>****
>
> </property>****
>
> ****
>
> After that, I typed dfsamdin command to check the file:****
>
> hdfs fsck /test3/****
>
> The result of above command:****
>
> /test3/hello005.txt: Under replicated
> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
> Target Replicas is 3 but found 2 replica(s).****
>
> Status: HEALTHY****
>
> Total size: 35 B****
>
> Total dirs: 1****
>
> Total files: 1****
>
> Total blocks (validated): 1 (avg. block size 35 B)****
>
> Minimally replicated blocks: 1 (100.0 %)****
>
> Over-replicated blocks: 0 (0.0 %)****
>
> Under-replicated blocks: 1 (100.0 %)****
>
> Mis-replicated blocks: 0 (0.0 %)****
>
> Default replication factor: 2****
>
> Average block replication: 2.0****
>
> Corrupt blocks: 0****
>
> Missing replicas: 1 (33.333332 %)****
>
> Number of data-nodes: 3****
>
> Number of racks: 1****
>
> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>
> ****
>
> ****
>
> Thanks,****
>
> Francis Hu****
>
> ****
>
> If I'm not mistaking "dfs.replication" parameter in config sets only
> default replication factor, which can be overrided when putting file to
> hdfs.****
>
>
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
Re: 答复: 答复: a question about dfs.replication
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Hu
It comes an point in my mind.And I tested it.
Before you set the dfs.replication to 2,the block may have already
existed with the original replication value 3.
After you changed the value, the replication of the former block was
still 3.
So the file that you created after the changement will be with the
replication 2.
However I do not have the access to change the replication within
the existed
files.
2013/7/2 Francis.Hu <fr...@reachjunction.com>
> Yes , it returns 2 correctly after "hdfs getconf -confkey
> dfs.replication"****
>
> ** **
>
> ****
>
> ** **
>
> but in web page ,it is 3 as below:****
>
> ****
>
> ** **
>
> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
> *����ʱ��:* Monday, July 01, 2013 23:24
> *�ռ���:* user@hadoop.apache.org
> *����:* Re: ��: a question about dfs.replication****
>
> ** **
>
> Hi ****
>
> ** **
>
> Could you please get the property value by using : hdfs getconf
> -confkey dfs.replication.****
>
>
> 发自我的 iPhone****
>
>
> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>
> ****
>
> Actually, My java client is running with the same configuration as the
> hadoop's . The dfs.replication is already set as 2 in my hadoop's
> configuration.****
>
> So i think the dfs.replication is already overrided by my configuration in
> hdfs-site.xml. but seems it doesn't work even i overrided the parameter
> evidently.****
>
> ****
>
> ****
>
> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>
> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
> *收件�?span lang="EN-US">:* user@hadoop.apache.org
> *主�**��:* Re: a question about dfs.replication****
>
> ****
>
> On 01.07.2013 10:19, Francis.Hu wrote:****
>
> Hi, All****
>
> ****
>
> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and
> two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After
> all configuration work is done, I started all nodes. Then I saved a file
> into HDFS through java client. nOW I can access hdfs web page:
> x.x.x.x:50070,and also see the file is already listed in the hdfs list.***
> *
>
> My question is: *The replication column in HDFS web page is showing as
> 3, not 2. Does anyone know What the problem is?*****
>
> * *****
>
> ---Actual setting of hdfs-site.xml****
>
> <property>****
>
> <name>dfs.replication</name>****
>
> <value>2</value>****
>
> </property>****
>
> ****
>
> After that, I typed dfsamdin command to check the file:****
>
> hdfs fsck /test3/****
>
> The result of above command:****
>
> /test3/hello005.txt: Under replicated
> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
> Target Replicas is 3 but found 2 replica(s).****
>
> Status: HEALTHY****
>
> Total size: 35 B****
>
> Total dirs: 1****
>
> Total files: 1****
>
> Total blocks (validated): 1 (avg. block size 35 B)****
>
> Minimally replicated blocks: 1 (100.0 %)****
>
> Over-replicated blocks: 0 (0.0 %)****
>
> Under-replicated blocks: 1 (100.0 %)****
>
> Mis-replicated blocks: 0 (0.0 %)****
>
> Default replication factor: 2****
>
> Average block replication: 2.0****
>
> Corrupt blocks: 0****
>
> Missing replicas: 1 (33.333332 %)****
>
> Number of data-nodes: 3****
>
> Number of racks: 1****
>
> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>
> ****
>
> ****
>
> Thanks,****
>
> Francis Hu****
>
> ****
>
> If I'm not mistaking "dfs.replication" parameter in config sets only
> default replication factor, which can be overrided when putting file to
> hdfs.****
>
>
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
reply: a question about dfs.replication
Posted by "Francis.Hu" <fr...@reachjunction.com>.
Thanks all of you, I just get the problem fixed through the command:
hdfs dfs -setrep -R -w 2 /
Is that an issue of HDFS ? Why do i need to execute manually a command to tell the hadoop the replication factor even it is set in hdfs-site.xml ?
Thanks,
Francis.Hu
发件人: Francis.Hu [mailto:francis.hu@reachjunction.com]
发送时间: Tuesday, July 02, 2013 11:30
收件人: user@hadoop.apache.org
主题: 答复: 答复: a question about dfs.replication
Yes , it returns 2 correctly after "hdfs getconf -confkey dfs.replication"
but in web page ,it is 3 as below:
发件人: yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
发送时间: Monday, July 01, 2013 23:24
收件人: user@hadoop.apache.org
主题: Re: 答复: a question about dfs.replication
Hi
Could you please get the property value by using : hdfs getconf -confkey dfs.replication.
鍙戣嚜鎴戠殑 iPhone
鍦?2013-7-1锛?5:51锛孎rancis.Hu <fr...@reachjunction.com> 鍐欓亾锛?br>
Actually, My java client is running with the same configuration as the hadoop's . The dfs.replication is already set as 2 in my hadoop's configuration.
So i think the dfs.replication is already overrided by my configuration in hdfs-site.xml. but seems it doesn't work even i overrided the parameter evidently.
鍙戜欢浜?span lang="EN-US">: 袝屑械谢褜褟薪芯胁 袘芯褉懈褋 [mailto:emelyanov@post.km.ru]
鍙戦€佹椂闂?span lang="EN-US">: Monday, July 01, 2013 15:18
鏀朵欢浜?span lang="EN-US">: user@hadoop.apache.org
涓婚: Re: a question about dfs.replication
On 01.07.2013 10:19, Francis.Hu wrote:
Hi, All
I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After all configuration work is done, I started all nodes. Then I saved a file into HDFS through java client. nOW I can access hdfs web page: x.x.x.x:50070,and also see the file is already listed in the hdfs list.
My question is: The replication column in HDFS web page is showing as 3, not 2. Does anyone know What the problem is?
---Actual setting of hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
After that, I typed dfsamdin command to check the file:
hdfs fsck /test3/
The result of above command:
/test3/hello005.txt: Under replicated BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006. Target Replicas is 3 but found 2 replica(s).
Status: HEALTHY
Total size: 35 B
Total dirs: 1
Total files: 1
Total blocks (validated): 1 (avg. block size 35 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 1 (100.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 1 (33.333332 %)
Number of data-nodes: 3
Number of racks: 1
FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds
Thanks,
Francis Hu
If I'm not mistaking "dfs.replication" parameter in config sets only default replication factor, which can be overrided when putting file to hdfs.
Re: 答复: 答复: a question about dfs.replication
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Hu
It comes an point in my mind.And I tested it.
Before you set the dfs.replication to 2,the block may have already
existed with the original replication value 3.
After you changed the value, the replication of the former block was
still 3.
So the file that you created after the changement will be with the
replication 2.
However I do not have the access to change the replication within
the existed
files.
2013/7/2 Francis.Hu <fr...@reachjunction.com>
> Yes , it returns 2 correctly after "hdfs getconf -confkey
> dfs.replication"****
>
> ** **
>
> ****
>
> ** **
>
> but in web page ,it is 3 as below:****
>
> ****
>
> ** **
>
> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
> *����ʱ��:* Monday, July 01, 2013 23:24
> *�ռ���:* user@hadoop.apache.org
> *����:* Re: ��: a question about dfs.replication****
>
> ** **
>
> Hi ****
>
> ** **
>
> Could you please get the property value by using : hdfs getconf
> -confkey dfs.replication.****
>
>
> 发自我的 iPhone****
>
>
> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>
> ****
>
> Actually, My java client is running with the same configuration as the
> hadoop's . The dfs.replication is already set as 2 in my hadoop's
> configuration.****
>
> So i think the dfs.replication is already overrided by my configuration in
> hdfs-site.xml. but seems it doesn't work even i overrided the parameter
> evidently.****
>
> ****
>
> ****
>
> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>
> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
> *收件�?span lang="EN-US">:* user@hadoop.apache.org
> *主�**��:* Re: a question about dfs.replication****
>
> ****
>
> On 01.07.2013 10:19, Francis.Hu wrote:****
>
> Hi, All****
>
> ****
>
> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and
> two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After
> all configuration work is done, I started all nodes. Then I saved a file
> into HDFS through java client. nOW I can access hdfs web page:
> x.x.x.x:50070,and also see the file is already listed in the hdfs list.***
> *
>
> My question is: *The replication column in HDFS web page is showing as
> 3, not 2. Does anyone know What the problem is?*****
>
> * *****
>
> ---Actual setting of hdfs-site.xml****
>
> <property>****
>
> <name>dfs.replication</name>****
>
> <value>2</value>****
>
> </property>****
>
> ****
>
> After that, I typed dfsamdin command to check the file:****
>
> hdfs fsck /test3/****
>
> The result of above command:****
>
> /test3/hello005.txt: Under replicated
> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
> Target Replicas is 3 but found 2 replica(s).****
>
> Status: HEALTHY****
>
> Total size: 35 B****
>
> Total dirs: 1****
>
> Total files: 1****
>
> Total blocks (validated): 1 (avg. block size 35 B)****
>
> Minimally replicated blocks: 1 (100.0 %)****
>
> Over-replicated blocks: 0 (0.0 %)****
>
> Under-replicated blocks: 1 (100.0 %)****
>
> Mis-replicated blocks: 0 (0.0 %)****
>
> Default replication factor: 2****
>
> Average block replication: 2.0****
>
> Corrupt blocks: 0****
>
> Missing replicas: 1 (33.333332 %)****
>
> Number of data-nodes: 3****
>
> Number of racks: 1****
>
> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>
> ****
>
> ****
>
> Thanks,****
>
> Francis Hu****
>
> ****
>
> If I'm not mistaking "dfs.replication" parameter in config sets only
> default replication factor, which can be overrided when putting file to
> hdfs.****
>
>
Re: 答复: 答复: a question about dfs.replication
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Hu
It comes an point in my mind.And I tested it.
Before you set the dfs.replication to 2,the block may have already
existed with the original replication value 3.
After you changed the value, the replication of the former block was
still 3.
So the file that you created after the changement will be with the
replication 2.
However I do not have the access to change the replication within
the existed
files.
2013/7/2 Francis.Hu <fr...@reachjunction.com>
> Yes , it returns 2 correctly after "hdfs getconf -confkey
> dfs.replication"****
>
> ** **
>
> ****
>
> ** **
>
> but in web page ,it is 3 as below:****
>
> ****
>
> ** **
>
> *������:* yypvsxf19870706 [mailto:yypvsxf19870706@gmail.com]
> *����ʱ��:* Monday, July 01, 2013 23:24
> *�ռ���:* user@hadoop.apache.org
> *����:* Re: ��: a question about dfs.replication****
>
> ** **
>
> Hi ****
>
> ** **
>
> Could you please get the property value by using : hdfs getconf
> -confkey dfs.replication.****
>
>
> 发自我的 iPhone****
>
>
> �?2013-7-1�?5:51，Francis.Hu <fr...@reachjunction.com> 写道�?br>****
>
> ****
>
> Actually, My java client is running with the same configuration as the
> hadoop's . The dfs.replication is already set as 2 in my hadoop's
> configuration.****
>
> So i think the dfs.replication is already overrided by my configuration in
> hdfs-site.xml. but seems it doesn't work even i overrided the parameter
> evidently.****
>
> ****
>
> ****
>
> *发件�**?span lang="EN-US">:* Емельянов Борис [mailto:emelyanov@post.km.ru<em...@post.km.ru>]
>
> *发�**�**�时�**?span lang="EN-US">:* Monday, July 01, 2013 15:18
> *收件�?span lang="EN-US">:* user@hadoop.apache.org
> *主�**��:* Re: a question about dfs.replication****
>
> ****
>
> On 01.07.2013 10:19, Francis.Hu wrote:****
>
> Hi, All****
>
> ****
>
> I am installing a cluster with Hadoop 2.0.5-alpha. I have one namenode and
> two datanodes. The dfs.replication is set as 2 in hdfs-site.xml. After
> all configuration work is done, I started all nodes. Then I saved a file
> into HDFS through java client. nOW I can access hdfs web page:
> x.x.x.x:50070,and also see the file is already listed in the hdfs list.***
> *
>
> My question is: *The replication column in HDFS web page is showing as
> 3, not 2. Does anyone know What the problem is?*****
>
> * *****
>
> ---Actual setting of hdfs-site.xml****
>
> <property>****
>
> <name>dfs.replication</name>****
>
> <value>2</value>****
>
> </property>****
>
> ****
>
> After that, I typed dfsamdin command to check the file:****
>
> hdfs fsck /test3/****
>
> The result of above command:****
>
> /test3/hello005.txt: Under replicated
> BP-609310498-192.168.219.129-1372323727200:blk_-1069303317294683372_1006.
> Target Replicas is 3 but found 2 replica(s).****
>
> Status: HEALTHY****
>
> Total size: 35 B****
>
> Total dirs: 1****
>
> Total files: 1****
>
> Total blocks (validated): 1 (avg. block size 35 B)****
>
> Minimally replicated blocks: 1 (100.0 %)****
>
> Over-replicated blocks: 0 (0.0 %)****
>
> Under-replicated blocks: 1 (100.0 %)****
>
> Mis-replicated blocks: 0 (0.0 %)****
>
> Default replication factor: 2****
>
> Average block replication: 2.0****
>
> Corrupt blocks: 0****
>
> Missing replicas: 1 (33.333332 %)****
>
> Number of data-nodes: 3****
>
> Number of racks: 1****
>
> FSCK ended at Sat Jun 29 16:51:37 CST 2013 in 6 milliseconds****
>
> ****
>
> ****
>
> Thanks,****
>
> Francis Hu****
>
> ****
>
> If I'm not mistaking "dfs.replication" parameter in config sets only
> default replication factor, which can be overrided when putting file to
> hdfs.****
>
>