You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Ravi Prakash (JIRA)" <ji...@apache.org> on 2017/06/22 20:03:00 UTC
[jira] [Comment Edited] (HDFS-12011) Add a new load balancing
volume choosing policy
[ https://issues.apache.org/jira/browse/HDFS-12011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059841#comment-16059841 ]
Ravi Prakash edited comment on HDFS-12011 at 6/22/17 8:02 PM:
--------------------------------------------------------------
Hi Chencan! Thanks for your contribution.
Technically this sounds like a good idea. Please bear in mind though that most often HDFS and YARN are running on the same node. Even if HDFS tracked which disks are being used accurately (I doubt it does), YARN may well be hammering the disks you choose using this policy. Moreover, we don't know what the Linux kernel's pagecache behavior really will be under the JVM (even when you think your write has completed, it may just be in the pagecache and will be synced to the block device on whenever Linux decides is a good time). Having said all of this, given that the policy is pluggable I can't think of a reason why we wouldn't want this policy. If you have production clusters (do you?) on which you can refine this policy, I think it'd be a great contribution. Have you done any performance tests that illustrate this policy is better than the other ones?
Also, its not clear to me that the reference count is a good proxy for measuring the load on a disk.
I'll take a look at the patch, but could you please try to fix all the -1s from HadoopQA's comment?
was (Author: raviprak):
Hi Chencan! Thanks for your contribution.
Technically this sounds like a good idea. Please bear in mind though that most often HDFS and YARN are running on the same node. Even if HDFS tracked which disks are being used accurately (I doubt it does), YARN may well be hammering the disks you choose using this policy. Moreover, we don't know what the Linux kernel's pagecache behavior really will be under the JVM (even when you think your write has completed, it may just be in the pagecache and will be synced to the block device on whenever Linux decides is a good time). Having said all of this, given that the policy is pluggable I can't think of a reason why we wouldn't want this policy. If you have production clusters (do you?) on which you can refine this policy, I think it'd be a great contribution. Have you done any performance tests that illustrate this policy is better than the other ones?
I'll take a look at the patch, but could you please try to fix all the -1s from HadoopQA's comment?
> Add a new load balancing volume choosing policy
> ------------------------------------------------
>
> Key: HDFS-12011
> URL: https://issues.apache.org/jira/browse/HDFS-12011
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Reporter: chencan
> Assignee: chencan
> Attachments: HADOOP-12011.patch
>
>
> There are two types of volume choosing policies when choose a volume inner a datanode to write in a datablock : RoundRobinVolumeChoosingPolicy and AvailableSpaceVolumeChoosingPolicy.This two policies do not take into account the fsvolume's load. We can add a new load balancing volume choosing policy, using existing reference in FsVolumeImpl.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org