You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Manoj Govindassamy (JIRA)" <ji...@apache.org> on 2017/01/31 18:58:51 UTC
[jira] (HDFS-11382) Persist Erasure Coding Policy ID in a new
optional field in INodeFile in FSImage
Manoj Govindassamy created HDFS-11382:
-----------------------------------------
Summary: Persist Erasure Coding Policy ID in a new optional field in INodeFile in FSImage
Key: HDFS-11382
URL: https://issues.apache.org/jira/browse/HDFS-11382
Project: Hadoop HDFS
Issue Type: Task
Components: hdfs
Affects Versions: 3.0.0-alpha1
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy
For Erasure Coded files, replication field in INodeFile message is re-used to store the EC Policy ID.
*FSDirWriteFileOp#addFile*
{noformat}
private static INodesInPath addFile(
FSDirectory fsd, INodesInPath existing, byte[] localName,
PermissionStatus permissions, short replication, long preferredBlockSize,
String clientName, String clientMachine)
throws IOException {
.. .. ..
try {
ErasureCodingPolicy ecPolicy = FSDirErasureCodingOp.
getErasureCodingPolicy(fsd.getFSNamesystem(), existing);
if (ecPolicy != null) {
replication = ecPolicy.getId(); <===
}
final BlockType blockType = ecPolicy != null?
BlockType.STRIPED : BlockType.CONTIGUOUS;
INodeFile newNode = newINodeFile(fsd.allocateNewInodeId(), permissions,
modTime, modTime, replication, preferredBlockSize, blockType);
newNode.setLocalName(localName);
newNode.toUnderConstruction(clientName, clientMachine);
newiip = fsd.addINode(existing, newNode, permissions.getPermission());
{noformat}
With HDFS-11268 fix, {{FSImageFormatPBINode#Loader#loadInodeFile}} is rightly getting the EC ID from the replication field and then uses the right Policy to construct the blocks.
*FSImageFormatPBINode#Loader#loadInodeFile*
{noformat}
ErasureCodingPolicy ecPolicy = (blockType == BlockType.STRIPED) ?
ErasureCodingPolicyManager.getPolicyByPolicyID((byte) replication) :
null;
{noformat}
The original intention was to re-use the replication field so the in-memory representation would be compact. But, this isn't necessary for the on-disk representation. replication is an optional field, and if we add another optional field for the EC policy, it won't be any extra space.
Also, we need to make sure to have the appropriate asserts in place to make sure both fields aren't set for the same INodeField.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org