You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Pol, Daniel" <da...@hpe.com> on 2017/07/14 21:51:38 UTC

How to persist custom EC policy with Hadoop 3.0 alpha4

Hi !

I'm testing  custom EC policies with alpha4 and I've successfully added and used a RS-5-2-256k EC policy.
The problem I have is that every time I restart HDFS all that info is lost and Namenode doesn't come up because it doesn't recognize the custom EC policy.
The relevant message from the Namenode log is:
2017-07-13 18:47:37,339 ERROR namenode.NameNode (NameNode.java:main(1709)) - Failed to start namenode.
java.lang.IllegalArgumentException: EC policy 'RS-5-2-256k' specified at dfs.namenode.ec.policies.enabled is not a valid policy. Please choose from list of available policies: [RS-6-3-64k, RS-3-2-64k, RS-LEGACY-6-3-64k, XOR-2-1-64k, RS-10-4-64k, ]
        at org.apache.hadoop.hdfs.server.namenode.ErasureCodingPolicyManager.init(ErasureCodingPolicyManager.java:123)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:868)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:693)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:664)
       at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:726)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:949)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:928)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1637)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1704)

Is there a place/setting where I can put my custom EC policies XML file so that it's loaded when HDFS starts so that the Namenode can recognize it and use it ?

Have a nice day,
Dani
"The more I learn, the less I know"