You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Huy Pham <ph...@yahoo-inc.com> on 2013/07/23 20:05:26 UTC
Get the tree structure of a HDFS dir, similar to dir/files
Hi All,
Do any of you have or can refer me to some sample Java code that get the tree structure of a HDFS directory, similar to the file system?
For example: I have a HDFS dir, called /data, inside data, there is /data/valid and /data/invalid, and so on, so I would need to be able to get the whole tree structure of that and know which is is a dir, which one is a file. Both program and HDFS are LOCAL.
In other words, what I look for is something similar to File class in Java, which has isDirectory() and list() to list all the children (files and dirs) of a dir. Found something in stackoverflow but it does not work.
Thanks
Huy
Re: Get the tree structure of a HDFS dir, similar to dir/files
Posted by Harsh J <ha...@cloudera.com>.
The FileSystem interface provides a recursive option for this. See
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean)
On Tue, Jul 23, 2013 at 11:35 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
> Do any of you have or can refer me to some sample Java code that get the
> tree structure of a HDFS directory, similar to the file system?
> For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
> In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files and
> dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
--
Harsh J
Re: Get the tree structure of a HDFS dir, similar to dir/files
Posted by Shahab Yunus <sh...@gmail.com>.
See this
https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
and
http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#isDirectory(org.apache.hadoop.fs.Path)
Basically you can write your own function, possible with recursion, to
iterate over a directory using the idea and combinations from above two
links.
Regards,
Shahab
On Tue, Jul 23, 2013 at 2:05 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
> Do any of you have or can refer me to some sample Java code that get
> the tree structure of a HDFS directory, similar to the file system?
> For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
> In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files
> and dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
>
Re: Get the tree structure of a HDFS dir, similar to dir/files
Posted by Shahab Yunus <sh...@gmail.com>.
See this
https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
and
http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#isDirectory(org.apache.hadoop.fs.Path)
Basically you can write your own function, possible with recursion, to
iterate over a directory using the idea and combinations from above two
links.
Regards,
Shahab
On Tue, Jul 23, 2013 at 2:05 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
> Do any of you have or can refer me to some sample Java code that get
> the tree structure of a HDFS directory, similar to the file system?
> For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
> In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files
> and dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
>
Re: Get the tree structure of a HDFS dir, similar to dir/files
Posted by Shahab Yunus <sh...@gmail.com>.
See this
https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
and
http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#isDirectory(org.apache.hadoop.fs.Path)
Basically you can write your own function, possible with recursion, to
iterate over a directory using the idea and combinations from above two
links.
Regards,
Shahab
On Tue, Jul 23, 2013 at 2:05 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
> Do any of you have or can refer me to some sample Java code that get
> the tree structure of a HDFS directory, similar to the file system?
> For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
> In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files
> and dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
>
Re: Get the tree structure of a HDFS dir, similar to dir/files
Posted by Shahab Yunus <sh...@gmail.com>.
See this
https://sites.google.com/site/hadoopandhive/home/how-to-read-all-files-in-a-directory-in-hdfs-using-hadoop-filesystem-api
and
http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html#isDirectory(org.apache.hadoop.fs.Path)
Basically you can write your own function, possible with recursion, to
iterate over a directory using the idea and combinations from above two
links.
Regards,
Shahab
On Tue, Jul 23, 2013 at 2:05 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
> Do any of you have or can refer me to some sample Java code that get
> the tree structure of a HDFS directory, similar to the file system?
> For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
> In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files
> and dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
>
Re: Get the tree structure of a HDFS dir, similar to dir/files
Posted by Harsh J <ha...@cloudera.com>.
The FileSystem interface provides a recursive option for this. See
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean)
On Tue, Jul 23, 2013 at 11:35 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
> Do any of you have or can refer me to some sample Java code that get the
> tree structure of a HDFS directory, similar to the file system?
> For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
> In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files and
> dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
--
Harsh J
Re: Get the tree structure of a HDFS dir, similar to dir/files
Posted by Pradeep Gollakota <pr...@gmail.com>.
Huy,
I think this question probably belongs in the Hadoop mailing list over the
Pig mailing list.
However, I think you're looking for
http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/fs/FileStatus.html
A FileStatus object can be acquired from a FileSystem object by calling the
.getFileStatus(Path path) method.
http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/fs/FileSystem.html
https://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/fs/Path.html
Hope this helps.
On Tue, Jul 23, 2013 at 2:05 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
> Do any of you have or can refer me to some sample Java code that get
> the tree structure of a HDFS directory, similar to the file system?
> For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
> In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files
> and dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
>
Re: Get the tree structure of a HDFS dir, similar to dir/files
Posted by Harsh J <ha...@cloudera.com>.
The FileSystem interface provides a recursive option for this. See
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean)
On Tue, Jul 23, 2013 at 11:35 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
> Do any of you have or can refer me to some sample Java code that get the
> tree structure of a HDFS directory, similar to the file system?
> For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
> In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files and
> dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
--
Harsh J
Re: Get the tree structure of a HDFS dir, similar to dir/files
Posted by Harsh J <ha...@cloudera.com>.
The FileSystem interface provides a recursive option for this. See
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean)
On Tue, Jul 23, 2013 at 11:35 PM, Huy Pham <ph...@yahoo-inc.com> wrote:
> Hi All,
> Do any of you have or can refer me to some sample Java code that get the
> tree structure of a HDFS directory, similar to the file system?
> For example: I have a HDFS dir, called /data, inside data, there is
> /data/valid and /data/invalid, and so on, so I would need to be able to get
> the whole tree structure of that and know which is is a dir, which one is a
> file. Both program and HDFS are LOCAL.
> In other words, what I look for is something similar to File class in
> Java, which has isDirectory() and list() to list all the children (files and
> dirs) of a dir. Found something in stackoverflow but it does not work.
> Thanks
> Huy
>
>
--
Harsh J