HDFS block replication problem

[Hendrik Haddorp <he...@gmx.net>]

Hi,

I have a HDFS test cluster with 7 data nodes and one name node. During 
write operations from Solr I get errors on the client side and messages 
like this in the name node logs:

017-11-27 11:16:09,130 DEBUG 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: 
Failed to choose from the next rack (location = /default/rack_02), retry 
choosing ramdomly

I also have entries like this:

2017-11-27 11:16:09,129 DEBUG 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: 
Failed to choose from local rack (location = /default/rack_01), retry 
with the rack of the next replica (location = /default/rack_02)
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException: 
[
Node /default/rack_01/10.37.131.184:50010 [
   Storage 
[DISK]DS-35c4ec83-f26f-44fb-a3be-21baf037357e:NORMAL:10.37.131.184:50010 
is not chosen since the node does not have enough DISK space 
(required=134217728, scheduled=14495514624, remaining=14205731794).
]

This seems to indicate that the disk on the data node is full. However 
when running "hdfs dfsadmin -report" I get the following for that node:

Name: 10.37.131.184:50010 (..........)
Hostname: ................
Rack: /default/rack_01
Decommission Status : Normal
Configured Capacity: 47766564864 (44.49 GB)
DFS Used: 4933282704 (4.59 GB)
Non DFS Used: 14173148272 (13.20 GB)
DFS Remaining: 28660133888 (26.69 GB)
DFS Used%: 10.33%
DFS Remaining%: 60.00%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Nov 27 11:52:59 EST 2017

According to that that should be plenty of space left. The web UI also 
reports:
Admin State    Capacity    Used    Non DFS Used    Remaining Blocks    
Block pool used    Failed Volumes    Version
In Service        44.49 GB  4.59 GB    13.17 GB        26.72 GB 
176852    4.59 GB (10.33%)          0                   2.7.3

Would be great if somebody could explain why my disk is reported to be full.

thanks,
Hendrik

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org