[jira] [Commented] (HDFS-14872) Read HDFS Blocks in Random Order

["Erik Krogen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942067#comment-16942067 ] 

Erik Krogen commented on HDFS-14872:
------------------------------------

+1 on this idea from me, we have considered building the same internally using the same pre-allocation strategy that you've described. For a large file with numerous blocks, it should be able to make a big difference.

> Read HDFS Blocks in Random Order
> --------------------------------
>
>                 Key: HDFS-14872
>                 URL: https://issues.apache.org/jira/browse/HDFS-14872
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: hdfs-client
>    Affects Versions: 2.8.5, 3.2.1
>            Reporter: David Mollitor
>            Priority: Major
>
> When the HDFS client is downloading (copying) an entire file, allow the client to download the blocks in random order.  If a lot of clients are reading the same file, in parallel, they will all download the first block, the second block, and so on, stampeding down the line.
> It would be interesting to spread the load across across all the available DataNodes.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org