[jira] [Commented] (NIFI-5452) Allow For Ignoring Block Locality When Writing to HDFS

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-5452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961486#comment-16961486 ] 

ASF subversion and git services commented on NIFI-5452:
-------------------------------------------------------

Commit dccbde40f3da5aecf8299513bcd4691702ffc02a in nifi's branch refs/heads/master from David Mollitor
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=dccbde4 ]

NIFI-5452: Allow ignore block locality in HDFS
Fixed formatting. Fallback to autoboxing

This closes #3652.


> Allow For Ignoring Block Locality When Writing to HDFS
> ------------------------------------------------------
>
>                 Key: NIFI-5452
>                 URL: https://issues.apache.org/jira/browse/NIFI-5452
>             Project: Apache NiFi
>          Issue Type: New Feature
>            Reporter: David Mollitor
>            Priority: Major
>             Fix For: 1.10.0
>
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> Now that [HDFS-13448] is available, add a new boolean configuration to the HDFS Sink configuration that enabled this.
> The basic issue is, as it currently stands, is the following:
> Imagine a cluster has four racks of hardware
> # Rack A is half management nodes and half datanodes
> # Rack B, C, D are all datanodes
> Now consider the following scenarios:
> If an instance of NiFi is located on a server outside of these racks, the data will be evenly distributed to each DataNode.
> If an instance of NiFi is running on Rack A, and is running co-located with a DataNode, then all of the HDFS Sink writes will first go to the local DataNode, thus overloading this single DataNode and filling it faster than all other DataNodes in the cluster.
> If an instance of NiFi is running on Rack A, on its own server, then all of the HDFS Sink writes will first go to a DataNode on Rack A, thus overloading the DataNodes on Rack A and filling those DataNodes faster than all other DataNodes in the cluster.  The issue here is compounded using many racks.  Rack A will always receive one copy of the each block, and the other two copies are scattered equally across the other racks.
> [HDFS-13448] adds a new flag to the HDFS client that requests to the NameNode that the first block should always be randomly placed.  Thus, if a NiFi instance is located on Rack A, the local node (or local rack) will not be overloaded.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)