You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by "Puri, Aseem" <As...@Honeywell.com> on 2009/05/21 07:39:00 UTC

More replication of map reduce output

Hi

            I have running a map reduce program on two node. My DFS
replication factor is one. Al files for input have one replication but
the output from reduce always have replication 3. Can anyone please tell
why it is so?

 

Thanks & Regards

Aseem Puri

RE: More replication of map reduce output

Posted by "Puri, Aseem" <As...@Honeywell.com>.

Yes I hav mark it as final. Now one more exception arises, my map reduce
program for word count is throwing exception.

09/05/21 11:31:37 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
09/05/21 11:31:37 INFO hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
/hadoop/mapred/system/job_200905211117_0003/job.jar could only be
replicated to 0 nodes, instead of 1
	at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F
SNamesystem.java:1280)
	at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:3
51)
	at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
	at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor
Impl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894)

	at org.apache.hadoop.ipc.Client.call(Client.java:697)
	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
	at $Proxy0.addBlock(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo
cationHandler.java:82)
	at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation
Handler.java:59)
	at $Proxy0.addBlock(Unknown Source)
	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DF
SClient.java:2814)
	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(D
FSClient.java:2696)
	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.j
ava:1996)
	at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli
ent.java:2183)

09/05/21 11:31:37 WARN hdfs.DFSClient: NotReplicatedYetException
sleeping /hadoop/mapred/system/job_200905211117_0003/job.jar retries
left 4

-----Original Message-----
From: Michael C. Toren [mailto:mct@toren.net] 
Sent: Thursday, May 21, 2009 11:27 AM
To: core-user@hadoop.apache.org
Subject: Re: More replication of map reduce output

On Thu, May 21, 2009 at 11:16:59AM +0530, Puri, Aseem wrote:
> I mean when my reduce tasks is set 1 part-00000 filw shows replication
facor as 3. But I set replication factor as 1 in hadoop-site.xml

Did you mark the replication factor configuration option as "final"?
e.g.:

    <property>
      <name>dfs.replication</name>
      <value>1</value>
      <final>true</final>
    </property>

-mct

Re: More replication of map reduce output

Posted by "Michael C. Toren" <mc...@toren.net>.

On Thu, May 21, 2009 at 11:16:59AM +0530, Puri, Aseem wrote:
> I mean when my reduce tasks is set 1 part-00000 filw shows replication facor as 3. But I set replication factor as 1 in hadoop-site.xml

Did you mark the replication factor configuration option as "final"?  e.g.:

    <property>
      <name>dfs.replication</name>
      <value>1</value>
      <final>true</final>
    </property>

-mct

RE: More replication of map reduce output

Posted by "Puri, Aseem" <As...@Honeywell.com>.

I mean when my reduce tasks is set 1 part-00000 filw shows replication facor as 3. But I set replication factor as 1 in hadoop-site.xml

-----Original Message-----
From: edward@udanax.org [mailto:edward@udanax.org] On Behalf Of Edward J. Yoon
Sent: Thursday, May 21, 2009 11:14 AM
To: core-user@hadoop.apache.org
Subject: Re: More replication of map reduce output

Do you mean the three files such as, part-00000? If so, you can set
the number of reduce tasks as 1.

On Thu, May 21, 2009 at 2:39 PM, Puri, Aseem <As...@honeywell.com> wrote:
> Hi
>
>            I have running a map reduce program on two node. My DFS
> replication factor is one. Al files for input have one replication but
> the output from reduce always have replication 3. Can anyone please tell
> why it is so?
>
>
>
> Thanks & Regards
>
> Aseem Puri
>
>
>
>



-- 
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org

Re: More replication of map reduce output

Posted by "Edward J. Yoon" <ed...@apache.org>.

Do you mean the three files such as, part-00000? If so, you can set
the number of reduce tasks as 1.

On Thu, May 21, 2009 at 2:39 PM, Puri, Aseem <As...@honeywell.com> wrote:
> Hi
>
>            I have running a map reduce program on two node. My DFS
> replication factor is one. Al files for input have one replication but
> the output from reduce always have replication 3. Can anyone please tell
> why it is so?
>
>
>
> Thanks & Regards
>
> Aseem Puri
>
>
>
>



-- 
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org