You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Akira Ajisaka <aa...@apache.org> on 2017/09/01 20:01:52 UTC
Re: Prime cause of NotEnoughReplicasException
Hi Nishant,
The debug message shows there are not enough racks configured to satisfy the rack awareness.
http://hadoop.apache.org/docs/r3.0.0-alpha4/hadoop-project-dist/hadoop-common/RackAwareness.html
If you don't need to place replicas in different racks, you can simply ignore the debug message.
Regards,
Akira
On 2017/08/01 11:53, Nishant Verma wrote:
> Hello
>
> On my namenode logs, I am seeing these error lines quite frequently. What is the reason behind this and how to overcome this? I have three data nodes and hdfs dfsadmin -report says 3 Live Datanodes.
>
> Kafka Connect is responsible of writing records from kafka topic to hdfs here. The data produce speed on kafka is high. 17 million records per hour. We have given a 30 minute rotate time interval to kafka connect to write the records on hdfs.
>
> 2017-08-01 09:12:55,215 DEBUG org.apache.hadoop.hdfs.StateChange: *BLOCK* NameNode.addBlock: file /topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp fileId=16394 for DFSClient_NONMAPREDUCE_-1139929652_127
> 2017-08-01 09:12:55,215 DEBUG org.apache.hadoop.hdfs.StateChange: BLOCK* getAdditionalBlock: /topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp inodeId 16394 for DFSClient_NONMAPREDUCE_-1139929652_127
> 2017-08-01 09:12:55,217 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to choose remote rack (location = ~/default-rack), fallback to local rack
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:702)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:582)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:350)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:216)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:113)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:128)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1567)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> 2017-08-01 09:12:55,220 DEBUG org.apache.hadoop.hdfs.StateChange: DIR* FSDirectory.addBlock: /topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp with blk_1073741825_1001 block is added to the in-memory file system
> 2017-08-01 09:12:55,220 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-09d1ec32-6d36-40ac-b558-f16289382d44:NORMAL:10.16.37.80:50010|RBW], ReplicaUC[[DISK]DS-73c57ed4-5b94-46d8-b3f5-70aaf2950319:NORMAL:10.16.37.140:50010|RBW]]} for /topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp
> 2017-08-01 09:12:55,220 DEBUG org.apache.hadoop.hdfs.StateChange: persistNewBlock: /topics/+tmp/testTopic/year=2017/month=08/day=01/hour=09/462eef10-37b6-4257-806b-45178886a886_tmp with new block blk_1073741825_1001{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-09d1ec32-6d36-40ac-b558-f16289382d44:NORMAL:10.16.37.80:50010|RBW], ReplicaUC[[DISK]DS-73c57ed4-5b94-46d8-b3f5-70aaf2950319:NORMAL:10.16.37.140:50010|RBW]]}, current total block count is 1
>
> Thanks
> Nishant Verma
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org