You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Zhe Zhang (JIRA)" <ji...@apache.org> on 2017/06/21 20:52:00 UTC
[jira] [Comment Edited] (HDFS-11634) Optimize BlockIterator when
iterating starts in the middle.
[ https://issues.apache.org/jira/browse/HDFS-11634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058203#comment-16058203 ]
Zhe Zhang edited comment on HDFS-11634 at 6/21/17 8:51 PM:
-----------------------------------------------------------
On one of our clusters with ~150 million blocks, {{getBlocks}} used to take 40ms. With the change it is taking 9ms, more than *4x* faster! This also leads to a significant improvement on NameNode RPC performance when Balancer is running.
was (Author: zhz):
On our cluster with ~150 million blocks, {{getBlocks}} used to take 40ms. With the change it is taking 9ms, more than *4x* faster! This also leads to a significant improvement on NameNode RPC performance when Balancer is running.
> Optimize BlockIterator when iterating starts in the middle.
> -----------------------------------------------------------
>
> Key: HDFS-11634
> URL: https://issues.apache.org/jira/browse/HDFS-11634
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.6.5
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
>
> Attachments: HDFS-11634.001.patch, HDFS-11634.002.patch, HDFS-11634.003.patch, HDFS-11634.004.patch, HDFS-11634.006.patch, HDFS-11634-branch-2.7.006.patch, HDFS-11643.005.patch, Screen Shot 2017-06-21 at 1.39.21 PM.png
>
>
> {{BlockManager.getBlocksWithLocations()}} needs to iterate blocks from a randomly selected {{startBlock}} index. It creates an iterator which points to the first block and then skips all blocks until {{startBlock}}. It is inefficient when DN has multiple storages. Instead of skipping blocks one by one we can skip entire storages. Should be more efficient on average.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org