You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Subroto <ss...@datameer.com> on 2012/06/19 10:29:23 UTC

DFSClient not able to write

Hi,

When DFS cluster; 
The datanodes keep on dumping the logs:
2012-06-19 04:05:04,842 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1233167859-10.10.35.8-1340093005960
2012-06-19 04:05:04,861 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 beginning handshake with NN
2012-06-19 04:05:04,979 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 successfully registered with NN
2012-06-19 04:05:04,980 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 using DELETEREPORT_INTERVAL of 300000 msec  BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000
2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 trying to claim ACTIVE state with txid=24
2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000
2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 1 msec to generate and 135 msecs for RPC and NN processing
2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@49d31859
2012-06-19 04:05:05,396 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block Verification Scanner initialized with interval 504 hours for block pool BP-1233167859-10.10.35.8-1340093005960.
2012-06-19 04:05:05,400 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added bpid=BP-1233167859-10.10.35.8-1340093005960 to blockPoolScannerMap, new size=1
2012-06-19 04:05:09,845 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:14,848 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:19,849 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:24,850 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:29,851 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:34,852 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:39,853 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:44,854 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:49,855 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%

The DFSClient operations are also not taking place properly:
2012-06-19 04:04:56,653 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
java.io.IOException: File /hbase/hbase.version could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1256)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1977)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:470)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:292)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42602)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)

Please let me know if there is any configuration which needs to be set…..
The hadoop version being used is: 2.0.0



Cheers,
Subroto Sanyal

Re: DFSClient not able to write

Posted by Subroto <ss...@datameer.com>.
Thanks Todd,

Yups… the problem was Disk Space….. I checked the NameNode UI which mentioned DFS Remaining as 0%

Is it possible to get some more detailed exception message in logs because, none of the logs (DataNode or Client) mention about this problem. ?
May be the NameNode UI also can indicate the same…..

Just like to know where the point:
> 5*BLOCK_SIZE
is documented….  :-(

Thanks again...

Cheers,
Subroto Sanyal

On Jun 19, 2012, at 6:22 PM, Todd Lipcon wrote:

> Hi Subroto,
> 
> Anything in the DN logs? Maybe the one DN is out of disk space? You
> need to have 5*BLOCK_SIZE free.
> 
> -Todd
> 
> On Tue, Jun 19, 2012 at 1:29 AM, Subroto <ss...@datameer.com> wrote:
>> Hi,
>> 
>> When DFS cluster;
>> The datanodes keep on dumping the logs:
>> 2012-06-19 04:05:04,842 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1233167859-10.10.35.8-1340093005960
>> 2012-06-19 04:05:04,861 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 beginning handshake with NN
>> 2012-06-19 04:05:04,979 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 successfully registered with NN
>> 2012-06-19 04:05:04,980 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 using DELETEREPORT_INTERVAL of 300000 msec  BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000
>> 2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 trying to claim ACTIVE state with txid=24
>> 2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000
>> 2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 1 msec to generate and 135 msecs for RPC and NN processing
>> 2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@49d31859
>> 2012-06-19 04:05:05,396 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block Verification Scanner initialized with interval 504 hours for block pool BP-1233167859-10.10.35.8-1340093005960.
>> 2012-06-19 04:05:05,400 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added bpid=BP-1233167859-10.10.35.8-1340093005960 to blockPoolScannerMap, new size=1
>> 2012-06-19 04:05:09,845 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:14,848 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:19,849 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:24,850 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:29,851 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:34,852 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:39,853 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:44,854 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:49,855 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 
>> The DFSClient operations are also not taking place properly:
>> 2012-06-19 04:04:56,653 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
>> java.io.IOException: File /hbase/hbase.version could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.
>>        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1256)
>>        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1977)
>>        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:470)
>>        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:292)
>>        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42602)
>>        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
>>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
>>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at javax.security.auth.Subject.doAs(Subject.java:396)
>>        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> 
>> Please let me know if there is any configuration which needs to be set…..
>> The hadoop version being used is: 2.0.0
>> 
>> 
>> 
>> Cheers,
>> Subroto Sanyal
> 
> 
> 
> -- 
> Todd Lipcon
> Software Engineer, Cloudera


Re: DFSClient not able to write

Posted by Todd Lipcon <to...@cloudera.com>.
Hi Subroto,

Anything in the DN logs? Maybe the one DN is out of disk space? You
need to have 5*BLOCK_SIZE free.

-Todd

On Tue, Jun 19, 2012 at 1:29 AM, Subroto <ss...@datameer.com> wrote:
> Hi,
>
> When DFS cluster;
> The datanodes keep on dumping the logs:
> 2012-06-19 04:05:04,842 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1233167859-10.10.35.8-1340093005960
> 2012-06-19 04:05:04,861 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 beginning handshake with NN
> 2012-06-19 04:05:04,979 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 successfully registered with NN
> 2012-06-19 04:05:04,980 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 using DELETEREPORT_INTERVAL of 300000 msec  BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000
> 2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 trying to claim ACTIVE state with txid=24
> 2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000
> 2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 1 msec to generate and 135 msecs for RPC and NN processing
> 2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@49d31859
> 2012-06-19 04:05:05,396 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block Verification Scanner initialized with interval 504 hours for block pool BP-1233167859-10.10.35.8-1340093005960.
> 2012-06-19 04:05:05,400 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added bpid=BP-1233167859-10.10.35.8-1340093005960 to blockPoolScannerMap, new size=1
> 2012-06-19 04:05:09,845 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:14,848 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:19,849 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:24,850 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:29,851 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:34,852 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:39,853 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:44,854 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:49,855 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>
> The DFSClient operations are also not taking place properly:
> 2012-06-19 04:04:56,653 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
> java.io.IOException: File /hbase/hbase.version could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.
>        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1256)
>        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1977)
>        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:470)
>        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:292)
>        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42602)
>        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
>        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
>        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
>        at java.security.AccessController.doPrivileged(Native Method)
>        at javax.security.auth.Subject.doAs(Subject.java:396)
>        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>
> Please let me know if there is any configuration which needs to be set…..
> The hadoop version being used is: 2.0.0
>
>
>
> Cheers,
> Subroto Sanyal



-- 
Todd Lipcon
Software Engineer, Cloudera