[jira] [Updated] (HDFS-6969) Archival Storage: INode#getStoragePolicyID should always return the latest storage policy

["Jing Zhao (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jing Zhao updated HDFS-6969:
----------------------------

    Attachment: HDFS-6969.000.patch

Initial patch. The main changes include:
1. Remove INode#getStoragePolicyID(int snapshotId). Instead, use the strategy described above
2. The current replication mechanism only checks the storage policy directly specified on the INodeFile. With the changes in #1, the storage policies specified on ancestral directories are also checked in case that no policy is specified on file.
3. Add several unit tests to verify the behavior when increasing replication factor with different storage policies. But looks like currently the replicas cannot be placed on the correct storages even if the block placement policy returns the correct results. Will continue digging.

> Archival Storage: INode#getStoragePolicyID should always return the latest storage policy
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-6969
>                 URL: https://issues.apache.org/jira/browse/HDFS-6969
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: balancer, namenode
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>         Attachments: HDFS-6969.000.patch
>
>
> In general, every file should only provide exact one storage policy for the Mover, no matter its snapshot states. Suppose a file /foo/bar, and it is contained in snapshots s1 and s2 of the root. If /foo/bar, /.snapshot/s1/foo/bar and /.snapshot/s2/foo/bar have different storage policies, when running Mover, we have to select one of the storage policies, among which the latest one should be the best. And if /foo/bar is deleted, we should still use its storage policy before the deletion, since the file deletion should not trigger data migration.
> Thus maybe what we can do is:
> 1. For a file with policy directly specified on it, alway follow the latest
> 2. Otherwise follow its latest parental path to identify its storage policy (simply following the parent link)



--
This message was sent by Atlassian JIRA
(v6.2#6252)