[jira] [Commented] (HADOOP-14210) Directories are not listed recursively when fs.defaultFs is viewFs

["Manoj Govindassamy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939330#comment-15939330 ] 

Manoj Govindassamy commented on HADOOP-14210:
---------------------------------------------

[~ajithshetty]
bq. {code}
-
-          result[i++] = new FileStatus(0, false, 0, 0,
+          boolean isDir=link.isMergeLink;
+          if (link.targetDirLinkList.length == 1) {
+            try {
+              isDir =
+                  link.targetFileSystem.getFileLinkStatus(link.getTargetLink())
+                      .isDirectory();
{code}
LinkMerge / MergeMounts are not supported yet. But, I see your point of reaching out to the target filesystem to find if the linked item is indeed a directory or not. 

[~xkrogen]
In the context of ViewFileSystem, all the linked entities - be it be Dir or File in target filesystem are of type INodeLink. Only the internal directories in the ViewFileSystem mount tree are of type INodeDir. So, [~ajithshetty] pointed out, ViewFileSystem treats only its internal directories as Dirs and all others as linked files. So, the FileStatus[] returned by ViewFileSystem has the Dir flag turned off for all the linked Directories in the target filesystem making the LS command stop the file tree traversal. 

Given the scale ViewFileSystem could be, returning millions of FileStatus[] across all namenodes could be a problem for Clients as well. So, I was assuming the intention for {{listStatus}} on ViewFileSystem is to only list the mount tree and not the entire world. But, this doesn't go well "ls -R" expectation from Clients. [~andrew.wang], any thoughts on the expectation for "ls -R" on ViewFileSystem root ?

> Directories are not listed recursively when fs.defaultFs is viewFs
> ------------------------------------------------------------------
>
>                 Key: HADOOP-14210
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14210
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: viewfs
>    Affects Versions: 2.7.0
>            Reporter: Ajith S
>              Labels: viewfs
>         Attachments: HDFS-8413.patch
>
>
> Mount a cluster on client throught viewFs mount table
> Example:
> {quote}
>  <property>
>     <name>fs.defaultFS</name>
>     <value>viewfs:///</value>
>   </property>
>     <property>
>         <name>fs.viewfs.mounttable.default.link./nn1</name>
>         <value>hdfs://ns1/</value>  <!-- HA nameservice -->
>     </property>
>     <property>
>         <name>fs.viewfs.mounttable.default.link./user</name>
>         <value>hdfs://host-72:8020/</value>
>     </property>
>  <property>
> {quote}
> Try to list the files recursively *(hdfs dfs -ls -R / or hadoop fs -ls -R /)* only the parent folders are listed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org