You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Nishant Verma <ni...@gmail.com> on 2017/08/01 09:53:44 UTC
Prime cause of NotEnoughReplicasException
Hello
On my namenode logs, I am seeing these error lines quite frequently. What
is the reason behind this and how to overcome this? I have three data nodes
and hdfs dfsadmin -report says 3 Live Datanodes.
Kafka Connect is responsible of writing records from kafka topic to hdfs
here. The data produce speed on kafka is high. 17 million records per hour.
We have given a 30 minute rotate time interval to kafka connect to write
the records on hdfs.
2017-08-01 09:12:55,215 DEBUG org.apache.hadoop.hdfs.StateChange: *BLOCK*
NameNode.addBlock: file
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
fileId=16394 for DFSClient_NONMAPREDUCE_-1139929652_127
2017-08-01 09:12:55,215 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK*
getAdditionalBlock:
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
inodeId 16394 for DFSClient_NONMAPREDUCE_-1139929652_127
2017-08-01 09:12:55,217 DEBUG
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed
to choose remote rack (location = ~/default-rack), fallback to local rack
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:702)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:582)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:350)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:216)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:113)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:128)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1567)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
2017-08-01 09:12:55,220 DEBUG org.apache.hadoop.hdfs.StateChange: DIR*
FSDirectory.addBlock:
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
with blk_1073741825_1001 block is added to the in-memory file system
2017-08-01 09:12:55,220 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocate blk_1073741825_1001{UCState=UNDER_CONSTRUCTION,
truncateBlock=null, primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-09d1ec32-6d36-40ac-b558-f16289382d44:NORMAL:10.16.37.80:50010|RBW],
ReplicaUC[[DISK]DS-73c57ed4-5b94-46d8-b3f5-70aaf2950319:NORMAL:10.16.37.140:50010|RBW]]}
for
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
2017-08-01 09:12:55,220 DEBUG org.apache.hadoop.hdfs.StateChange:
persistNewBlock:
/topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
with new block blk_1073741825_1001{UCState=UNDER_CONSTRUCTION,
truncateBlock=null, primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-09d1ec32-6d36-40ac-b558-f16289382d44:NORMAL:10.16.37.80:50010|RBW],
ReplicaUC[[DISK]DS-73c57ed4-5b94-46d8-b3f5-70aaf2950319:NORMAL:10.16.37.140:50010|RBW]]},
current total block count is 1
Thanks
Nishant Verma
Re: Prime cause of NotEnoughReplicasException
Posted by Akira Ajisaka <aa...@apache.org>.
Hi Nishant,
The debug message shows there are not enough racks configured to satisfy the rack awareness.
http://hadoop.apache.org/docs/r3.0.0-alpha4/hadoop-project-dist/hadoop-common/RackAwareness.html
If you don't need to place replicas in different racks, you can simply ignore the debug message.
Regards,
Akira
On 2017/08/01 11:53, Nishant Verma wrote:
> Hello
>
> On my namenode logs, I am seeing these error lines quite frequently. What is the reason behind this and how to overcome this? I have three data nodes and hdfs dfsadmin -report says 3 Live Datanodes.
>
> Kafka Connect is responsible of writing records from kafka topic to hdfs here. The data produce speed on kafka is high. 17 million records per hour. We have given a 30 minute rotate time interval to kafka connect to write the records on hdfs.
>
> 2017-08-01 09:12:55,215 DEBUG org.apache.hadoop.hdfs.StateChange: *BLOCK* NameNode.addBlock: file /topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp fileId=16394 for DFSClient_NONMAPREDUCE_-1139929652_127
> 2017-08-01 09:12:55,215 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK* getAdditionalBlock: /topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp inodeId 16394 for DFSClient_NONMAPREDUCE_-1139929652_127
> 2017-08-01 09:12:55,217 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to choose remote rack (location = ~/default-rack), fallback to local rack
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:702)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:582)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:350)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:216)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:113)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:128)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1567)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> 2017-08-01 09:12:55,220 DEBUG org.apache.hadoop.hdfs.StateChange: DIR* FSDirectory.addBlock: /topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp with blk_1073741825_1001 block is added to the in-memory file system
> 2017-08-01 09:12:55,220 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-09d1ec32-6d36-40ac-b558-f16289382d44:NORMAL:10.16.37.80:50010|RBW], ReplicaUC[[DISK]DS-73c57ed4-5b94-46d8-b3f5-70aaf2950319:NORMAL:10.16.37.140:50010|RBW]]} for /topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
> 2017-08-01 09:12:55,220 DEBUG org.apache.hadoop.hdfs.StateChange: persistNewBlock: /topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp with new block blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-09d1ec32-6d36-40ac-b558-f16289382d44:NORMAL:10.16.37.80:50010|RBW], ReplicaUC[[DISK]DS-73c57ed4-5b94-46d8-b3f5-70aaf2950319:NORMAL:10.16.37.140:50010|RBW]]}, current total block count is 1
>
> Thanks
> Nishant Verma
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org