You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Ankit Bhatnagar <an...@yahoo.com> on 2013/05/16 23:55:44 UTC
recursive list in java without block
Hi folks,
How can I get a recursive listing of file using java code from HDFS (hadoop 0.23.7*)
i mean equivalent to ls -R?
Ankit
Re: recursive list in java without block
Posted by Harsh J <ha...@cloudera.com>.
Hi Kun,
So I think I was incorrect as I only checked the 0.20/1.0 APIs.
We do have optimal recursive listing support in 2.0 for both
FileContext and FileSystem APIs:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean).
I'd encourage using this, rather than FsShell which is not a public
API for users and may change its behavior anytime in future (no
guarantee).
On Fri, May 17, 2013 at 2:48 PM, Kun Ling <lk...@gmail.com> wrote:
> Hi Ankit,
>
> Following Harsh's advice, I have found out that: although none of the
> FileSystem.java and DistributedFileSystem.java have support recursively
> liststatus(). However, the FsShell.java did have an ls() method which is
> used to support hadoop command like lsr ( that is the ls -R in Linux).
>
>
> yours,
>
> Kun Ling
>
>
> On Fri, May 17, 2013 at 6:59 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> The FileSystem API doesn't provide a utility to do recursive listing
>> yet, so you'd have to build it on your own.
>>
>> MR and the Fs Shell, both do seem to have inbuilt support for such a
>> utility though.
>>
>> On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
>> <an...@yahoo.com> wrote:
>> > Hi folks,
>> >
>> > How can I get a recursive listing of file using java code from HDFS
>> > (hadoop
>> > 0.23.7*)
>> > i mean equivalent to ls -R?
>> >
>> > Ankit
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> http://www.lingcc.com
--
Harsh J
Re: recursive list in java without block
Posted by Harsh J <ha...@cloudera.com>.
Hi Kun,
So I think I was incorrect as I only checked the 0.20/1.0 APIs.
We do have optimal recursive listing support in 2.0 for both
FileContext and FileSystem APIs:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean).
I'd encourage using this, rather than FsShell which is not a public
API for users and may change its behavior anytime in future (no
guarantee).
On Fri, May 17, 2013 at 2:48 PM, Kun Ling <lk...@gmail.com> wrote:
> Hi Ankit,
>
> Following Harsh's advice, I have found out that: although none of the
> FileSystem.java and DistributedFileSystem.java have support recursively
> liststatus(). However, the FsShell.java did have an ls() method which is
> used to support hadoop command like lsr ( that is the ls -R in Linux).
>
>
> yours,
>
> Kun Ling
>
>
> On Fri, May 17, 2013 at 6:59 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> The FileSystem API doesn't provide a utility to do recursive listing
>> yet, so you'd have to build it on your own.
>>
>> MR and the Fs Shell, both do seem to have inbuilt support for such a
>> utility though.
>>
>> On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
>> <an...@yahoo.com> wrote:
>> > Hi folks,
>> >
>> > How can I get a recursive listing of file using java code from HDFS
>> > (hadoop
>> > 0.23.7*)
>> > i mean equivalent to ls -R?
>> >
>> > Ankit
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> http://www.lingcc.com
--
Harsh J
Re: recursive list in java without block
Posted by Harsh J <ha...@cloudera.com>.
Hi Kun,
So I think I was incorrect as I only checked the 0.20/1.0 APIs.
We do have optimal recursive listing support in 2.0 for both
FileContext and FileSystem APIs:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean).
I'd encourage using this, rather than FsShell which is not a public
API for users and may change its behavior anytime in future (no
guarantee).
On Fri, May 17, 2013 at 2:48 PM, Kun Ling <lk...@gmail.com> wrote:
> Hi Ankit,
>
> Following Harsh's advice, I have found out that: although none of the
> FileSystem.java and DistributedFileSystem.java have support recursively
> liststatus(). However, the FsShell.java did have an ls() method which is
> used to support hadoop command like lsr ( that is the ls -R in Linux).
>
>
> yours,
>
> Kun Ling
>
>
> On Fri, May 17, 2013 at 6:59 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> The FileSystem API doesn't provide a utility to do recursive listing
>> yet, so you'd have to build it on your own.
>>
>> MR and the Fs Shell, both do seem to have inbuilt support for such a
>> utility though.
>>
>> On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
>> <an...@yahoo.com> wrote:
>> > Hi folks,
>> >
>> > How can I get a recursive listing of file using java code from HDFS
>> > (hadoop
>> > 0.23.7*)
>> > i mean equivalent to ls -R?
>> >
>> > Ankit
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> http://www.lingcc.com
--
Harsh J
Re: recursive list in java without block
Posted by Harsh J <ha...@cloudera.com>.
Hi Kun,
So I think I was incorrect as I only checked the 0.20/1.0 APIs.
We do have optimal recursive listing support in 2.0 for both
FileContext and FileSystem APIs:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.html#listFiles(org.apache.hadoop.fs.Path,%20boolean).
I'd encourage using this, rather than FsShell which is not a public
API for users and may change its behavior anytime in future (no
guarantee).
On Fri, May 17, 2013 at 2:48 PM, Kun Ling <lk...@gmail.com> wrote:
> Hi Ankit,
>
> Following Harsh's advice, I have found out that: although none of the
> FileSystem.java and DistributedFileSystem.java have support recursively
> liststatus(). However, the FsShell.java did have an ls() method which is
> used to support hadoop command like lsr ( that is the ls -R in Linux).
>
>
> yours,
>
> Kun Ling
>
>
> On Fri, May 17, 2013 at 6:59 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> The FileSystem API doesn't provide a utility to do recursive listing
>> yet, so you'd have to build it on your own.
>>
>> MR and the Fs Shell, both do seem to have inbuilt support for such a
>> utility though.
>>
>> On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
>> <an...@yahoo.com> wrote:
>> > Hi folks,
>> >
>> > How can I get a recursive listing of file using java code from HDFS
>> > (hadoop
>> > 0.23.7*)
>> > i mean equivalent to ls -R?
>> >
>> > Ankit
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> http://www.lingcc.com
--
Harsh J
Re: recursive list in java without block
Posted by Kun Ling <lk...@gmail.com>.
Hi Ankit,
Following Harsh's advice, I have found out that: although none of the
FileSystem.java and DistributedFileSystem.java have support recursively
liststatus(). However, the FsShell.java did have an ls() method which is
used to support hadoop command like lsr ( that is the ls -R in Linux).
yours,
Kun Ling
On Fri, May 17, 2013 at 6:59 AM, Harsh J <ha...@cloudera.com> wrote:
> The FileSystem API doesn't provide a utility to do recursive listing
> yet, so you'd have to build it on your own.
>
> MR and the Fs Shell, both do seem to have inbuilt support for such a
> utility though.
>
> On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
> <an...@yahoo.com> wrote:
> > Hi folks,
> >
> > How can I get a recursive listing of file using java code from HDFS
> (hadoop
> > 0.23.7*)
> > i mean equivalent to ls -R?
> >
> > Ankit
>
>
>
> --
> Harsh J
>
--
http://www.lingcc.com
Re: recursive list in java without block
Posted by Kun Ling <lk...@gmail.com>.
Hi Ankit,
Following Harsh's advice, I have found out that: although none of the
FileSystem.java and DistributedFileSystem.java have support recursively
liststatus(). However, the FsShell.java did have an ls() method which is
used to support hadoop command like lsr ( that is the ls -R in Linux).
yours,
Kun Ling
On Fri, May 17, 2013 at 6:59 AM, Harsh J <ha...@cloudera.com> wrote:
> The FileSystem API doesn't provide a utility to do recursive listing
> yet, so you'd have to build it on your own.
>
> MR and the Fs Shell, both do seem to have inbuilt support for such a
> utility though.
>
> On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
> <an...@yahoo.com> wrote:
> > Hi folks,
> >
> > How can I get a recursive listing of file using java code from HDFS
> (hadoop
> > 0.23.7*)
> > i mean equivalent to ls -R?
> >
> > Ankit
>
>
>
> --
> Harsh J
>
--
http://www.lingcc.com
Re: recursive list in java without block
Posted by Kun Ling <lk...@gmail.com>.
Hi Ankit,
Following Harsh's advice, I have found out that: although none of the
FileSystem.java and DistributedFileSystem.java have support recursively
liststatus(). However, the FsShell.java did have an ls() method which is
used to support hadoop command like lsr ( that is the ls -R in Linux).
yours,
Kun Ling
On Fri, May 17, 2013 at 6:59 AM, Harsh J <ha...@cloudera.com> wrote:
> The FileSystem API doesn't provide a utility to do recursive listing
> yet, so you'd have to build it on your own.
>
> MR and the Fs Shell, both do seem to have inbuilt support for such a
> utility though.
>
> On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
> <an...@yahoo.com> wrote:
> > Hi folks,
> >
> > How can I get a recursive listing of file using java code from HDFS
> (hadoop
> > 0.23.7*)
> > i mean equivalent to ls -R?
> >
> > Ankit
>
>
>
> --
> Harsh J
>
--
http://www.lingcc.com
Re: recursive list in java without block
Posted by Kun Ling <lk...@gmail.com>.
Hi Ankit,
Following Harsh's advice, I have found out that: although none of the
FileSystem.java and DistributedFileSystem.java have support recursively
liststatus(). However, the FsShell.java did have an ls() method which is
used to support hadoop command like lsr ( that is the ls -R in Linux).
yours,
Kun Ling
On Fri, May 17, 2013 at 6:59 AM, Harsh J <ha...@cloudera.com> wrote:
> The FileSystem API doesn't provide a utility to do recursive listing
> yet, so you'd have to build it on your own.
>
> MR and the Fs Shell, both do seem to have inbuilt support for such a
> utility though.
>
> On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
> <an...@yahoo.com> wrote:
> > Hi folks,
> >
> > How can I get a recursive listing of file using java code from HDFS
> (hadoop
> > 0.23.7*)
> > i mean equivalent to ls -R?
> >
> > Ankit
>
>
>
> --
> Harsh J
>
--
http://www.lingcc.com
Re: recursive list in java without block
Posted by Harsh J <ha...@cloudera.com>.
The FileSystem API doesn't provide a utility to do recursive listing
yet, so you'd have to build it on your own.
MR and the Fs Shell, both do seem to have inbuilt support for such a
utility though.
On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
<an...@yahoo.com> wrote:
> Hi folks,
>
> How can I get a recursive listing of file using java code from HDFS (hadoop
> 0.23.7*)
> i mean equivalent to ls -R?
>
> Ankit
--
Harsh J
Re: recursive list in java without block
Posted by Harsh J <ha...@cloudera.com>.
The FileSystem API doesn't provide a utility to do recursive listing
yet, so you'd have to build it on your own.
MR and the Fs Shell, both do seem to have inbuilt support for such a
utility though.
On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
<an...@yahoo.com> wrote:
> Hi folks,
>
> How can I get a recursive listing of file using java code from HDFS (hadoop
> 0.23.7*)
> i mean equivalent to ls -R?
>
> Ankit
--
Harsh J
Re: recursive list in java without block
Posted by Harsh J <ha...@cloudera.com>.
The FileSystem API doesn't provide a utility to do recursive listing
yet, so you'd have to build it on your own.
MR and the Fs Shell, both do seem to have inbuilt support for such a
utility though.
On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
<an...@yahoo.com> wrote:
> Hi folks,
>
> How can I get a recursive listing of file using java code from HDFS (hadoop
> 0.23.7*)
> i mean equivalent to ls -R?
>
> Ankit
--
Harsh J
Re: recursive list in java without block
Posted by Harsh J <ha...@cloudera.com>.
The FileSystem API doesn't provide a utility to do recursive listing
yet, so you'd have to build it on your own.
MR and the Fs Shell, both do seem to have inbuilt support for such a
utility though.
On Fri, May 17, 2013 at 3:25 AM, Ankit Bhatnagar
<an...@yahoo.com> wrote:
> Hi folks,
>
> How can I get a recursive listing of file using java code from HDFS (hadoop
> 0.23.7*)
> i mean equivalent to ls -R?
>
> Ankit
--
Harsh J