You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Huy Pham <ph...@yahoo-inc.com> on 2013/07/23 20:05:26 UTC

Get the tree structure of a HDFS dir, similar to dir/files

Hi All,
   Do any of you have or can refer me to some sample Java code that get the tree structure of a HDFS directory, similar to the file system?
   For example: I have a HDFS dir, called /data, inside data, there is /data/valid and /data/invalid, and so on, so I would need to be able to get the whole tree structure of that and know which is is a dir, which one is a file. Both program and HDFS are LOCAL.
   In other words, what I look for is something similar to File class in Java, which has isDirectory() and list() to list all the children (files and dirs) of a dir. Found something in stackoverflow but it does not work.
Thanks
Huy



Re: Get the tree structure of a HDFS dir, similar to dir/files

Posted by Harsh J <ha...@cloudera.com>.
The FileSystem interface provides a recursive option for this. See
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean)

On Tue, Jul 23, 2013 at 11:35 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
>    Do any of you have or can refer me to some sample Java code that get the
> tree structure of a HDFS directory, similar to the file system?
>    For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
>    In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files and
> dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>



-- 
Harsh J

Re: Get the tree structure of a HDFS dir, similar to dir/files

Posted by Shahab Yunus <sh...@gmail.com>.
See this
https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api

and

http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#isDirectory(org.apache.hadoop.fs.Path)

Basically you can write your own function, possible with recursion, to
iterate over a directory using the idea and combinations from above two
links.

Regards,
Shahab


On Tue, Jul 23, 2013 at 2:05 PM, Huy Pham <ph...@yahoo-inc.com> wrote:

>  Hi All,
>    Do any of you have or can refer me to some sample Java code that get
> the tree structure of a HDFS directory, similar to the file system?
>    For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
>    In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files
> and dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
>

Re: Get the tree structure of a HDFS dir, similar to dir/files

Posted by Shahab Yunus <sh...@gmail.com>.
See this
https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api

and

http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#isDirectory(org.apache.hadoop.fs.Path)

Basically you can write your own function, possible with recursion, to
iterate over a directory using the idea and combinations from above two
links.

Regards,
Shahab


On Tue, Jul 23, 2013 at 2:05 PM, Huy Pham <ph...@yahoo-inc.com> wrote:

>  Hi All,
>    Do any of you have or can refer me to some sample Java code that get
> the tree structure of a HDFS directory, similar to the file system?
>    For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
>    In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files
> and dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
>

Re: Get the tree structure of a HDFS dir, similar to dir/files

Posted by Shahab Yunus <sh...@gmail.com>.
See this
https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api

and

http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#isDirectory(org.apache.hadoop.fs.Path)

Basically you can write your own function, possible with recursion, to
iterate over a directory using the idea and combinations from above two
links.

Regards,
Shahab


On Tue, Jul 23, 2013 at 2:05 PM, Huy Pham <ph...@yahoo-inc.com> wrote:

>  Hi All,
>    Do any of you have or can refer me to some sample Java code that get
> the tree structure of a HDFS directory, similar to the file system?
>    For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
>    In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files
> and dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
>

Re: Get the tree structure of a HDFS dir, similar to dir/files

Posted by Shahab Yunus <sh...@gmail.com>.
See this
https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api

and

http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#isDirectory(org.apache.hadoop.fs.Path)

Basically you can write your own function, possible with recursion, to
iterate over a directory using the idea and combinations from above two
links.

Regards,
Shahab


On Tue, Jul 23, 2013 at 2:05 PM, Huy Pham <ph...@yahoo-inc.com> wrote:

>  Hi All,
>    Do any of you have or can refer me to some sample Java code that get
> the tree structure of a HDFS directory, similar to the file system?
>    For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
>    In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files
> and dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
>

Re: Get the tree structure of a HDFS dir, similar to dir/files

Posted by Harsh J <ha...@cloudera.com>.
The FileSystem interface provides a recursive option for this. See
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean)

On Tue, Jul 23, 2013 at 11:35 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
>    Do any of you have or can refer me to some sample Java code that get the
> tree structure of a HDFS directory, similar to the file system?
>    For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
>    In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files and
> dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>



-- 
Harsh J

Re: Get the tree structure of a HDFS dir, similar to dir/files

Posted by Pradeep Gollakota <pr...@gmail.com>.
Huy,

I think this question probably belongs in the Hadoop mailing list over the
Pig mailing list.
However, I think you're looking for
http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/fs/FileStatus.html
A FileStatus object can be acquired from a FileSystem object by calling the
.getFileStatus(Path path) method.

http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/fs/FileSystem.html
https://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/fs/Path.html

Hope this helps.



On Tue, Jul 23, 2013 at 2:05 PM, Huy Pham <ph...@yahoo-inc.com> wrote:

> Hi All,
>    Do any of you have or can refer me to some sample Java code that get
> the tree structure of a HDFS directory, similar to the file system?
>    For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
>    In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files
> and dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
>

Re: Get the tree structure of a HDFS dir, similar to dir/files

Posted by Harsh J <ha...@cloudera.com>.
The FileSystem interface provides a recursive option for this. See
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean)

On Tue, Jul 23, 2013 at 11:35 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
>    Do any of you have or can refer me to some sample Java code that get the
> tree structure of a HDFS directory, similar to the file system?
>    For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
>    In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files and
> dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>



-- 
Harsh J

Re: Get the tree structure of a HDFS dir, similar to dir/files

Posted by Harsh J <ha...@cloudera.com>.
The FileSystem interface provides a recursive option for this. See
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean)

On Tue, Jul 23, 2013 at 11:35 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
>    Do any of you have or can refer me to some sample Java code that get the
> tree structure of a HDFS directory, similar to the file system?
>    For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
>    In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files and
> dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>



-- 
Harsh J