[jira] [Commented] (HDFS-13910) Improve BlockPlacementPolicy for small clusters

["SAKTHIVEL (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715898#comment-16715898 ] 

SAKTHIVEL commented on HDFS-13910:
----------------------------------

Hi,

I am facing same issue in a 4 data node cluster running on docker container.  This issue seems matching to my environment. I am facing this issue when i tried to write a small hello world file in hdfs through scala or java program.

But local dfs client command copyFromLocal works without any issues. Please share any work around for this issue. 

 

[hdfs@hadoop-master logs]$ hadoop version
Hadoop 2.9.1
Subversion https://github.com/apache/hadoop.git -r e30710aea4e6e55e69372929106cf119af06fd0e
Compiled by root on 2018-04-16T09:33Z
Compiled with protoc 2.5.0
From source with checksum 7d6d2b655115c6cc336d662cc2b919bd
This command was run using /opt/yarn/hadoop-2.9.1/share/hadoop/common/hadoop-common-2.9.1.jar
[hdfs@hadoop-master logs]$

> Improve BlockPlacementPolicy for small clusters
> -----------------------------------------------
>
>                 Key: HDFS-13910
>                 URL: https://issues.apache.org/jira/browse/HDFS-13910
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>            Reporter: Shweta
>            Assignee: Shweta
>            Priority: Critical
>         Attachments: HDFS-13910.001.patch
>
>
> From investigations and a few test occurrences, the NameNode BlockPlacementPolicy’s considerLoad can be bad for small test clusters.
> A small (1-node) cluster may trigger a corner case of maxLoad = 0. In this case, filtering should not take place. 
> When there is no heartbeat, maxLoad will be 0 and the existing logic will filter out many nodes that have workload, which should be avoided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org