You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by abhay ratnaparkhi <ab...@gmail.com> on 2011/05/02 20:40:45 UTC

hi

Hello,

I have installed using 0.20.2 in cygwin environment.
I'm getting following error while trying to put some content on HDFS.
Please suggest any solution.

Abhay
*
$ bin/hadoop dfs -put ../../mylogs/other_logs/access_log-tommy.091210
/input/i1*

11/05/02 23:59:47 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /input/i1
could only be replicated to 0 nodes, instead of 1
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:599)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at
java.security.AccessController.doPrivileged(AccessController.java:284
)
        at javax.security.auth.Subject.doAs(Subject.java:573)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

        at org.apache.hadoop.ipc.Client.call(Client.java:740)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
        at $Proxy0.addBlock(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:45)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:599)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI
nvocationHandler.java:82)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat
ionHandler.java:59)
        at $Proxy0.addBlock(Unknown Source)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock
(DFSClient.java:2937)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStrea
m(DFSClient.java:2819)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClien
t.java:2102)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFS
Client.java:2288)

RE: hi

Posted by "Habermaas, William" <Wi...@fatwire.com>.
Look at your namenode log.  From the log info you supplied one possibility is that HDFS doesn't think there is any space available on your system. Some CYGWIN installs do not work properly because BASH doesn't work when called internally by Hadoop.  HDFS depends on running shell commands like DF to discover if there is space on the system.  

Bill 

-----Original Message-----
From: abhay ratnaparkhi [mailto:abhay.ratnaparkhi@gmail.com] 
Sent: Monday, May 02, 2011 2:41 PM
To: common-user@hadoop.apache.org
Subject: hi

Hello,

I have installed using 0.20.2 in cygwin environment.
I'm getting following error while trying to put some content on HDFS.
Please suggest any solution.

Abhay
*
$ bin/hadoop dfs -put ../../mylogs/other_logs/access_log-tommy.091210
/input/i1*

11/05/02 23:59:47 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /input/i1
could only be replicated to 0 nodes, instead of 1
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:599)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at
java.security.AccessController.doPrivileged(AccessController.java:284
)
        at javax.security.auth.Subject.doAs(Subject.java:573)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

        at org.apache.hadoop.ipc.Client.call(Client.java:740)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
        at $Proxy0.addBlock(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:45)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:599)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI
nvocationHandler.java:82)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat
ionHandler.java:59)
        at $Proxy0.addBlock(Unknown Source)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock
(DFSClient.java:2937)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStrea
m(DFSClient.java:2819)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClien
t.java:2102)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFS
Client.java:2288)

Re: hi

Posted by abhay ratnaparkhi <ab...@gmail.com>.
I resolved all the exceptions in starting namenode/datanodes.
Now I'm getting following exception.

$ bin/hadoop dfs -copyFromLocal ../../mylogs/other_logs/access_log-tommy.091210
 /i3
11/05/03 02:35:02 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor excepti
on  for block blk_-8156764341648560739_1001java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:191)
        at java.io.DataInputStream.readLong(DataInputStream.java:410)
        at org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.read
Fields(DataTransferProtocol.java:119)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.ru
n(DFSClient.java:2424)

11/05/03 02:35:02 WARN hdfs.DFSClient: Error Recovery for block blk_-81567643416
48560739_1001 bad datanode[0] 127.0.0.1:50010
copyFromLocal: All datanodes 127.0.0.1:50010 are bad. Aborting...
11/05/03 02:35:02 ERROR hdfs.DFSClient: Exception closing file /i3 : java.io.IOE
xception: All datanodes 127.0.0.1:50010 are bad. Aborting...
java.io.IOException: All datanodes 127.0.0.1:50010 are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError
(DFSClient.java:2556)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClien
t.java:2102)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFS
Client.java:2265)


On 5/3/11, Harsh J <ha...@cloudera.com> wrote:
> The FAQ knows too:
> http://wiki.apache.org/hadoop/FAQ#What_does_.22file_could_only_be_replicated_to_0_nodes.2C_instead_of_1.22_mean.3F
> :)
>
> On Tue, May 3, 2011 at 12:10 AM, abhay ratnaparkhi
> <ab...@gmail.com> wrote:
>> Hello,
>>
>> I have installed using 0.20.2 in cygwin environment.
>> I'm getting following error while trying to put some content on HDFS.
>> Please suggest any solution.
>>
>> Abhay
>> *
>> $ bin/hadoop dfs -put ../../mylogs/other_logs/access_log-tommy.091210
>> /input/i1*
>>
>> 11/05/02 23:59:47 WARN hdfs.DFSClient: DataStreamer Exception:
>> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /input/i1
>> could only be replicated to 0 nodes, instead of 1
>>        at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
>>        at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>>        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>> sorImpl.java:37)
>>        at java.lang.reflect.Method.invoke(Method.java:599)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>>        at
>> java.security.AccessController.doPrivileged(AccessController.java:284
>> )
>>        at javax.security.auth.Subject.doAs(Subject.java:573)
>>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>>
>>        at org.apache.hadoop.ipc.Client.call(Client.java:740)
>>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>>        at $Proxy0.addBlock(Unknown Source)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>> java:45)
>>        at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>> sorImpl.java:37)
>>        at java.lang.reflect.Method.invoke(Method.java:599)
>>        at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI
>> nvocationHandler.java:82)
>>        at
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat
>> ionHandler.java:59)
>>        at $Proxy0.addBlock(Unknown Source)
>>        at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock
>> (DFSClient.java:2937)
>>        at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStrea
>> m(DFSClient.java:2819)
>>        at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClien
>> t.java:2102)
>>        at
>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFS
>> Client.java:2288)
>>
>
>
>
> --
> Harsh J
>

Re: hi

Posted by Harsh J <ha...@cloudera.com>.
The FAQ knows too:
http://wiki.apache.org/hadoop/FAQ#What_does_.22file_could_only_be_replicated_to_0_nodes.2C_instead_of_1.22_mean.3F
:)

On Tue, May 3, 2011 at 12:10 AM, abhay ratnaparkhi
<ab...@gmail.com> wrote:
> Hello,
>
> I have installed using 0.20.2 in cygwin environment.
> I'm getting following error while trying to put some content on HDFS.
> Please suggest any solution.
>
> Abhay
> *
> $ bin/hadoop dfs -put ../../mylogs/other_logs/access_log-tommy.091210
> /input/i1*
>
> 11/05/02 23:59:47 WARN hdfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /input/i1
> could only be replicated to 0 nodes, instead of 1
>        at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
>        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:37)
>        at java.lang.reflect.Method.invoke(Method.java:599)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
>        at
> java.security.AccessController.doPrivileged(AccessController.java:284
> )
>        at javax.security.auth.Subject.doAs(Subject.java:573)
>        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
>
>        at org.apache.hadoop.ipc.Client.call(Client.java:740)
>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
>        at $Proxy0.addBlock(Unknown Source)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:45)
>        at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:37)
>        at java.lang.reflect.Method.invoke(Method.java:599)
>        at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI
> nvocationHandler.java:82)
>        at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat
> ionHandler.java:59)
>        at $Proxy0.addBlock(Unknown Source)
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock
> (DFSClient.java:2937)
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStrea
> m(DFSClient.java:2819)
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClien
> t.java:2102)
>        at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFS
> Client.java:2288)
>



-- 
Harsh J

Re: hi

Posted by Bharath Mundlapudi <bh...@yahoo.com>.
The key message is in the first line. 


11/05/02 23:59:47 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /input/i1
could only be replicated to 0 nodes, instead of 1

Zero datanodes were found to perform this operation. Check the DN logs.

-Bharath



----- Original Message -----
From: abhay ratnaparkhi <ab...@gmail.com>
To: common-user@hadoop.apache.org
Cc: 
Sent: Monday, May 2, 2011 11:40 AM
Subject: Re: hi

Hello,

I have installed using 0.20.2 in cygwin environment.
I'm getting following error while trying to put some content on HDFS.
Please suggest any solution.

Abhay
*
$ bin/hadoop dfs -put ../../mylogs/other_logs/access_log-tommy.091210
/input/i1*

11/05/02 23:59:47 WARN hdfs.DFSClient: DataStreamer Exception:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /input/i1
could only be replicated to 0 nodes, instead of 1
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1271)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:599)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
        at
java.security.AccessController.doPrivileged(AccessController.java:284
)
        at javax.security.auth.Subject.doAs(Subject.java:573)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

        at org.apache.hadoop.ipc.Client.call(Client.java:740)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
        at $Proxy0.addBlock(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:45)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:599)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI
nvocationHandler.java:82)
        at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat
ionHandler.java:59)
        at $Proxy0.addBlock(Unknown Source)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock
(DFSClient.java:2937)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStrea
m(DFSClient.java:2819)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClien
t.java:2102)
        at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFS
Client.java:2288)