[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages multiple erasure coding policies

["Zhe Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127149#comment-15127149 ] 

Zhe Zhang commented on HDFS-7866:
---------------------------------

Thanks Rui for the work. I think HDFS-9658 is almost ready.

I think the below issues are worth more discussions:
# Under HDFS-8833 we [discussed | https://issues.apache.org/jira/browse/HDFS-8833?focusedCommentId=14718396&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14718396] the idea of encoding {{replication}} and {{ecPolicy}} together. Let's revisit the discussion and reach an agreement here. [~jingzhao], [~andrew.wang], [~walter.k.su], would be great to have your opinions here.
{code}
// For erasure coded files, replication is used to store ec policy id
...
  public short getErasureCodingPolicyID() {
    if (isStriped()) {
      return HeaderFormat.getReplication(header);
    }
    return -1;
  }
{code}
Rather than directly reusing the {{REPLICATION}} section, I think we should combine the current {{REPLICATION}} and {{IS_STRIPED}} sections and give it a meaningful name. Something like {{BLOCK_LAYOUT_AND_REDUNDANCY}}. Then we have decoding methods to retrieve replication factor, EC policy, and so forth from it. I think 12 bits is enough to encode all possible layout/redundancy policies. For contiguous blocks, we have 11 bits for repl factor, up to 2048. For striped blocks, we will possibly support replication factor in the future. At that time we can have 4 bits for EC policy (16) and 7 for repl factor (128).
# About policy IDs, should we simplify the mapping with a similar approach as {{BlockStoragePolicySuite}}?

> Erasure coding: NameNode manages multiple erasure coding policies
> -----------------------------------------------------------------
>
>                 Key: HDFS-7866
>                 URL: https://issues.apache.org/jira/browse/HDFS-7866
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Kai Zheng
>            Assignee: Rui Li
>         Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch, HDFS-7866-v3.patch, HDFS-7866.4.patch, HDFS-7866.5.patch, HDFS-7866.6.patch, HDFS-7866.7.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in authorized and controlled approach. The provided facilities will be used to implement DFSAdmin commands so admin can list available EC schemas, then could choose some of them for target EC zones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)