[jira] [Created] (HDFS-13720) HDFS dataset Anti-Affinity Block Placement across DataNodes for data local task optimization (improve Spark executor utilization & performance)

["Hari Sekhon (JIRA)" <ji...@apache.org>]

Hari Sekhon created HDFS-13720:
----------------------------------

             Summary: HDFS dataset Anti-Affinity Block Placement across DataNodes for data local task optimization (improve Spark executor utilization & performance)
                 Key: HDFS-13720
                 URL: https://issues.apache.org/jira/browse/HDFS-13720
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: balancer & mover, block placement, performance
    Affects Versions: 2.7.3
         Environment: Hortonworks HDP 2.6
            Reporter: Hari Sekhon


Improvement Request for Anti-Affinity Block Placement across datanodes such that for a given data set the blocks are distributed evenly across all available datanodes in order to improve task scheduling while maintaining data locality.

This could be done via a client side write flag as well as via a balancer command switch combined with giving a target path to files or directories to redistributed as evenly as possible across all datanodes in the cluster.

See this following Spark issue which causes massive under-utilisation across jobs. Only 30-50% of executor cores were being used for tasks due to data locality targeting. Many executors doing literally nothing, while holding significant cluster resources, because the data set, which in at least one job was large enough to have 30,000 tasks churning though slowly on only a subset of the available executors. The workaround in the end was to disable data local tasks in Spark, but if everyone did that the bottleneck would go back to being the network and it undermines Hadoop's first premise of don't move the data to compute. For performance critical jobs, returning tasks to Yarn isn't a good idea either, they want the jobs to use all the resources available, not just the resources on a subset of nodes that hold a given dataset or pulling half the blocks across the network.

https://issues.apache.org/jira/browse/SPARK-24474



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org