[jira] [Commented] (HDFS-13724) Storage Tiering Show Paths with Policies applied

["Hari Sekhon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534859#comment-16534859 ] 

Hari Sekhon commented on HDFS-13724:
------------------------------------

I tried a workaround of dupming the for now is to do the following as hdfs superuser - dump the fsimage, convert to XML and then grep the tiers path info:
{code:java}
su - hdfs
kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs
hdfs dfsadmin -fetchImage .
# this step might take a long time on big clusters (eg. 20 mins for 12GB fsimage.xml result file from a moderate 600TB cluster)
hadoop oiv -i $(ls -tr fsimage_* | tail -n1) -p XML -o fsimage.xml
grep ...{code}
but I can't find anything relating to 'policy' or the name of our storage policy or the directory I know it's applied to.

> Storage Tiering Show Paths with Policies applied
> ------------------------------------------------
>
>                 Key: HDFS-13724
>                 URL: https://issues.apache.org/jira/browse/HDFS-13724
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Hari Sekhon
>            Priority: Major
>
> Improvement Request to add an hdfs storagepolicies command to find paths for which storage policies have been applied.
> Right now you must explicitly query a single directory to get its policy to determine if one has been applied, but if another hadoop admin has configured policies on anything but trivially obvious paths such as /archive then there is no way to find which paths have policies applied to them other than by querying every single directory and subdirectory one by one which might potentially have a policy, eg:
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir2
> hdfs storagepolicies -getStoragePolicy -path /dir3
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir2
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir3
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN/subdirN/subsubdirN
> ...{code}
> In my current environment for example, a policy was configured for /data/blah which doesn't show when trying
> {code:java}
>  hdfs storagepolicies -getStoragePolicy -path /data{code}
> and I had no way of knowing that I had to do:
> {code:java}
>  hdfs storagepolicies -getStoragePolicy -path /data/blah{code}
> other than trial and error of trying every directory and every subdirectory in hdfs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org