You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Virajith Jalaparti (JIRA)" <ji...@apache.org> on 2016/07/16 00:31:20 UTC
[jira] [Comment Edited] (HDFS-10637) Modifications to remove the
assumption that FsVolumes are backed by java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380362#comment-15380362 ]
Virajith Jalaparti edited comment on HDFS-10637 at 7/16/16 12:30 AM:
---------------------------------------------------------------------
The patch contains the following changes (text is incorporated from comments in HDFS-9809):
# Instead of associating an FsVolume with a base path (which is a {{java.io.File}}), we associate it with a {{StorageLocation}}. This allows removing the dependence on {{java.io.File}} and replacing it with a more general one, which can point to a {{java.io.File}} or an abstract {{URI}} representing an external storage. Using {{StorageLocation}} instead of defining a new type for location allows us to reuse its functionality and plug into the rest of the code easily. Following this intuition, we replaced {{FsVolumeSpi#getBasePath}} with {{FsVolumeSpi#getStorageLocation}}. As a result, comparisons and references to FsVolumes which were done using the {{java.io.File}} returned by {{FsVolumeSpi#getBasePath}} are now replaced by comparisons and references to the {{StorageLocation}} returned by {{FsVolumeSpi#getStorageLocation}}.
# Similarly, the patch also replaces calls to get*Dir on FsVolumeImpl. In particular, the {{DirectoryScanner.ReportCompiler}} calls on {{FsVolumeSpi#getFinalizedDir}} and compiles the report assuming that this returns a {{java.io.File}}. However, in HDFS-9806, data may not be stored in files. Further, the {{DirectoryScanner.ReportCompiler#compileReport}} function assumes the way blocks are stored in FsVolumes which can be different for different {{FsVolumeSpi}} implementations. To address these assumptions and to hide the details on how volumes implement block storage, {{ReportCompiler#compileReport}} is moved to {{FsVolumeSpi}}.
# A {{FsVolumeImplBuilder}} is added and calls to the constructor of {{FsVolumeImpl}} are replaced with those to the builder. The idea behind this is to construct the appropriate volume based on the {{StorageLocation}} (specified using {{dfs.datanode.data.dir}}). For example, as part of HDFS-9806, if a {{StorageLocation}} is of a PROVIDED type, we would construct a {{ProvidedVolumeImpl}}. Otherwise, a {{FsVolumeImpl}} would be built.
was (Author: virajith):
The patch contains the following changes (some text is incorporated from comments in HDFS-9809):
# Instead of associating an FsVolume with a base path (which is a {{java.io.File}}), we associate it with a {{StorageLocation}}. This allows removing the dependence on {{java.io.File}} and replacing it with a more general one, which can point to a {{java.io.File}} or an abstract {{URI}} representing an external storage. Using {{StorageLocation}} instead of defining a new type for location allows us to reuse its functionality and plug into the rest of the code easily. Following this intuition, we replaced {{FsVolumeSpi#getBasePath}} with {{FsVolumeSpi#getStorageLocation}}. As a result, comparisons and references to FsVolumes which were done using the {{java.io.File}} returned by {{FsVolumeSpi#getBasePath}} are now replaced by comparisons and references to the {{StorageLocation}} returned by {{FsVolumeSpi#getStorageLocation}}.
# Similarly, the patch also replaces calls to get*Dir on FsVolumeImpl. In particular, the {{DirectoryScanner.ReportCompiler}} calls on {{FsVolumeSpi#getFinalizedDir}} and compiles the report assuming that this returns a {{java.io.File}}. However, in HDFS-9806, data may not be stored in files. Further, the {{DirectoryScanner.ReportCompiler#compileReport}} function assumes the way blocks are stored in FsVolumes which can be different for different {{FsVolumeSpi}} implementations. To address these assumptions and to hide the details on how volumes implement block storage, {{ReportCompiler#compileReport}} is moved to {{FsVolumeSpi}}.
# A {{FsVolumeImplBuilder}} is added and calls to the constructor of {{FsVolumeImpl}} are replaced with those to the builder. The idea behind this is to construct the appropriate volume based on the {{StorageLocation}} (specified using {{dfs.datanode.data.dir}}). For example, as part of HDFS-9806, if a {{StorageLocation}} is of a PROVIDED type, we would construct a {{ProvidedVolumeImpl}}. Otherwise, a {{FsVolumeImpl}} would be built.
> Modifications to remove the assumption that FsVolumes are backed by java.io.File.
> ---------------------------------------------------------------------------------
>
> Key: HDFS-10637
> URL: https://issues.apache.org/jira/browse/HDFS-10637
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
> Reporter: Virajith Jalaparti
> Attachments: HDFS-10637.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org