You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Subroto <ss...@datameer.com> on 2012/06/19 10:29:23 UTC
DFSClient not able to write
Hi,
When DFS cluster;
The datanodes keep on dumping the logs:
2012-06-19 04:05:04,842 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1233167859-10.10.35.8-1340093005960
2012-06-19 04:05:04,861 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 beginning handshake with NN
2012-06-19 04:05:04,979 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 successfully registered with NN
2012-06-19 04:05:04,980 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 using DELETEREPORT_INTERVAL of 300000 msec BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000
2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 trying to claim ACTIVE state with txid=24
2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000
2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 1 msec to generate and 135 msecs for RPC and NN processing
2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@49d31859
2012-06-19 04:05:05,396 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block Verification Scanner initialized with interval 504 hours for block pool BP-1233167859-10.10.35.8-1340093005960.
2012-06-19 04:05:05,400 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added bpid=BP-1233167859-10.10.35.8-1340093005960 to blockPoolScannerMap, new size=1
2012-06-19 04:05:09,845 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:14,848 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:19,849 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:24,850 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:29,851 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:34,852 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:39,853 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:44,854 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:49,855 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
The DFSClient operations are also not taking place properly:
2012-06-19 04:04:56,653 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
java.io.IOException: File /hbase/hbase.version could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1256)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1977)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:470)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:292)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42602)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
Please let me know if there is any configuration which needs to be set…..
The hadoop version being used is: 2.0.0
Cheers,
Subroto Sanyal
Re: DFSClient not able to write
Posted by Subroto <ss...@datameer.com>.
Thanks Todd,
Yups… the problem was Disk Space….. I checked the NameNode UI which mentioned DFS Remaining as 0%
Is it possible to get some more detailed exception message in logs because, none of the logs (DataNode or Client) mention about this problem. ?
May be the NameNode UI also can indicate the same…..
Just like to know where the point:
> 5*BLOCK_SIZE
is documented…. :-(
Thanks again...
Cheers,
Subroto Sanyal
On Jun 19, 2012, at 6:22 PM, Todd Lipcon wrote:
> Hi Subroto,
>
> Anything in the DN logs? Maybe the one DN is out of disk space? You
> need to have 5*BLOCK_SIZE free.
>
> -Todd
>
> On Tue, Jun 19, 2012 at 1:29 AM, Subroto <ss...@datameer.com> wrote:
>> Hi,
>>
>> When DFS cluster;
>> The datanodes keep on dumping the logs:
>> 2012-06-19 04:05:04,842 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1233167859-10.10.35.8-1340093005960
>> 2012-06-19 04:05:04,861 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 beginning handshake with NN
>> 2012-06-19 04:05:04,979 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 successfully registered with NN
>> 2012-06-19 04:05:04,980 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 using DELETEREPORT_INTERVAL of 300000 msec BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000
>> 2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 trying to claim ACTIVE state with txid=24
>> 2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000
>> 2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 1 msec to generate and 135 msecs for RPC and NN processing
>> 2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@49d31859
>> 2012-06-19 04:05:05,396 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block Verification Scanner initialized with interval 504 hours for block pool BP-1233167859-10.10.35.8-1340093005960.
>> 2012-06-19 04:05:05,400 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added bpid=BP-1233167859-10.10.35.8-1340093005960 to blockPoolScannerMap, new size=1
>> 2012-06-19 04:05:09,845 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:14,848 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:19,849 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:24,850 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:29,851 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:34,852 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:39,853 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:44,854 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>> 2012-06-19 04:05:49,855 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>>
>> The DFSClient operations are also not taking place properly:
>> 2012-06-19 04:04:56,653 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
>> java.io.IOException: File /hbase/hbase.version could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation.
>> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1256)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1977)
>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:470)
>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:292)
>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42602)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>>
>> Please let me know if there is any configuration which needs to be set…..
>> The hadoop version being used is: 2.0.0
>>
>>
>>
>> Cheers,
>> Subroto Sanyal
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
Re: DFSClient not able to write
Posted by Todd Lipcon <to...@cloudera.com>.
Hi Subroto,
Anything in the DN logs? Maybe the one DN is out of disk space? You
need to have 5*BLOCK_SIZE free.
-Todd
On Tue, Jun 19, 2012 at 1:29 AM, Subroto <ss...@datameer.com> wrote:
> Hi,
>
> When DFS cluster;
> The datanodes keep on dumping the logs:
> 2012-06-19 04:05:04,842 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1233167859-10.10.35.8-1340093005960
> 2012-06-19 04:05:04,861 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 beginning handshake with NN
> 2012-06-19 04:05:04,979 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 successfully registered with NN
> 2012-06-19 04:05:04,980 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 using DELETEREPORT_INTERVAL of 300000 msec BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000
> 2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 trying to claim ACTIVE state with txid=24
> 2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000
> 2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 1 msec to generate and 135 msecs for RPC and NN processing
> 2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@49d31859
> 2012-06-19 04:05:05,396 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block Verification Scanner initialized with interval 504 hours for block pool BP-1233167859-10.10.35.8-1340093005960.
> 2012-06-19 04:05:05,400 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added bpid=BP-1233167859-10.10.35.8-1340093005960 to blockPoolScannerMap, new size=1
> 2012-06-19 04:05:09,845 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:14,848 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:19,849 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:24,850 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:29,851 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:34,852 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:39,853 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:44,854 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
> 2012-06-19 04:05:49,855 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
>
> The DFSClient operations are also not taking place properly:
> 2012-06-19 04:04:56,653 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
> java.io.IOException: File /hbase/hbase.version could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation.
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1256)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1977)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:470)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:292)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42602)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>
> Please let me know if there is any configuration which needs to be set…..
> The hadoop version being used is: 2.0.0
>
>
>
> Cheers,
> Subroto Sanyal
--
Todd Lipcon
Software Engineer, Cloudera