[jira] [Commented] (HDFS-7613) Block placement policy for erasure coding groups

["Ming Ma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14629070#comment-14629070 ] 

Ming Ma commented on HDFS-7613:
-------------------------------

Interesting work. There are couple issues around the extensibility of block placement policies. They aren't EC specific; so we don't have to tackle them here. But we would like to raise them here if it helps to make future refactoring easier.

* Balancer and Mover have built-in assumption it is the default block placement policy. So every time we have a new block placement policy, we need to modify those tools. One suggestion mentioned in HDFS-1431 is to run Balancer and Mover inside NN.
* BlockManager has built-in assumption about rack policy in functions such as useDelHint, blockHasEnoughRacks. That means when we have new block placement policy, we need to modify BlockManager to account for the new policy. HDFS-8647 should improve that.
* Ability to reuse or compose new policies based on existing block placement policies. This is different from HDFS-4894 to support different policies for different files. For example, HDFS-7541 adds upgrade domain policy. It will be nice if we can support both upgrade domain policy and EC policy without any code change at run time for a given file. https://issues.apache.org/jira/secure/attachment/12687808/SupportforfastHDFSdatanoderollingupgrade.pdf's "Support for non­topology based policy" section suggested a more flexible API.
* As we have new policies, migration from old policy to new policy on production clusters becomes necessary. [~ctrezzo] has worked in this area and plan to open a new jira for that.


> Block placement policy for erasure coding groups
> ------------------------------------------------
>
>                 Key: HDFS-7613
>                 URL: https://issues.apache.org/jira/browse/HDFS-7613
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Walter Su
>         Attachments: HDFS-7613.001.patch
>
>
> Blocks in an erasure coding group should be placed in different failure domains -- different DataNodes at the minimum, and different racks ideally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)