You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Libo Yu <yu...@hotmail.com> on 2014/03/28 03:03:51 UTC
How to get locations of blocks programmatically?
Hi all,
"hadoop path fsck -files -block -locations" can list locations for all blocks in the path.
Is it possible to list all blocks and the block locations for a given path programmatically?
Thanks,
Libo
RE: How to get locations of blocks programmatically?
Posted by Libo Yu <yu...@hotmail.com>.
Thanks, it is useful. If I know the block I'd, how to find out its location?
From: wheeleast@gmail.com
Date: Fri, 28 Mar 2014 10:12:10 +0800
Subject: Re: How to get locations of blocks programmatically?
To: user@hadoop.apache.org
Hi Libo,DFSClient.getBlockLocations, is this what you want?
Regards,Wangda Tan
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
Hi all,
"hadoop path fsck -files -block -locations" can list locations for all blocks in the path.
Is it possible to list all blocks and the block locations for a given path programmatically?
Thanks,
Libo
RE: How to get locations of blocks programmatically?
Posted by Libo Yu <yu...@hotmail.com>.
Thanks, it is useful. If I know the block I'd, how to find out its location?
From: wheeleast@gmail.com
Date: Fri, 28 Mar 2014 10:12:10 +0800
Subject: Re: How to get locations of blocks programmatically?
To: user@hadoop.apache.org
Hi Libo,DFSClient.getBlockLocations, is this what you want?
Regards,Wangda Tan
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
Hi all,
"hadoop path fsck -files -block -locations" can list locations for all blocks in the path.
Is it possible to list all blocks and the block locations for a given path programmatically?
Thanks,
Libo
RE: How to get locations of blocks programmatically?
Posted by Libo Yu <yu...@hotmail.com>.
Thanks, it is useful. If I know the block I'd, how to find out its location?
From: wheeleast@gmail.com
Date: Fri, 28 Mar 2014 10:12:10 +0800
Subject: Re: How to get locations of blocks programmatically?
To: user@hadoop.apache.org
Hi Libo,DFSClient.getBlockLocations, is this what you want?
Regards,Wangda Tan
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
Hi all,
"hadoop path fsck -files -block -locations" can list locations for all blocks in the path.
Is it possible to list all blocks and the block locations for a given path programmatically?
Thanks,
Libo
RE: How to get locations of blocks programmatically?
Posted by Libo Yu <yu...@hotmail.com>.
Thanks, it is useful. If I know the block I'd, how to find out its location?
From: wheeleast@gmail.com
Date: Fri, 28 Mar 2014 10:12:10 +0800
Subject: Re: How to get locations of blocks programmatically?
To: user@hadoop.apache.org
Hi Libo,DFSClient.getBlockLocations, is this what you want?
Regards,Wangda Tan
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
Hi all,
"hadoop path fsck -files -block -locations" can list locations for all blocks in the path.
Is it possible to list all blocks and the block locations for a given path programmatically?
Thanks,
Libo
Re: How to get locations of blocks programmatically?
Posted by Wangda Tan <wh...@gmail.com>.
Hi Libo,
DFSClient.getBlockLocations, is this what you want?
Regards,
Wangda Tan
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>
Re: How to get locations of blocks programmatically?
Posted by Harsh J <ha...@cloudera.com>.
Yes, use http://hadoop.apache.org/docs/stable2/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.Path,
long, long)
On Fri, Mar 28, 2014 at 7:33 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
--
Harsh J
Re: How to get locations of blocks programmatically?
Posted by Hardik Pandya <sm...@gmail.com>.
have you looked into FileSystem API this is hadoop v2.2.0
http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html
does not exist in
http://hadoop.apache.org/docs/r1.2.0/api/org/apache/hadoop/fs/FileSystem.html
org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/LocatedFileStatus.html>
> *listFiles
<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html#listFiles%28org.apache.hadoop.fs.Path,%20boolean%29>*
(Path<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/Path.html>
f,
boolean recursive)
List the statuses and block locations of the files in the given
path. org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/LocatedFileStatus.html>
> *listLocatedStatus
<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html#listLocatedStatus%28org.apache.hadoop.fs.Path%29>*
(Path<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/Path.html>
f)
List the statuses of the files/directories in the given path if
the path is a directory.
On Thu, Mar 27, 2014 at 10:03 PM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>
Re: How to get locations of blocks programmatically?
Posted by Stanley Shi <ss...@gopivotal.com>.
FileSystem.getFileBlockLocations(...)
Regards,
*Stanley Shi,*
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>
Re: How to get locations of blocks programmatically?
Posted by Wangda Tan <wh...@gmail.com>.
Hi Libo,
DFSClient.getBlockLocations, is this what you want?
Regards,
Wangda Tan
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>
Re: How to get locations of blocks programmatically?
Posted by Stanley Shi <ss...@gopivotal.com>.
FileSystem.getFileBlockLocations(...)
Regards,
*Stanley Shi,*
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>
Re: How to get locations of blocks programmatically?
Posted by Wangda Tan <wh...@gmail.com>.
Hi Libo,
DFSClient.getBlockLocations, is this what you want?
Regards,
Wangda Tan
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>
Re: How to get locations of blocks programmatically?
Posted by Harsh J <ha...@cloudera.com>.
Yes, use http://hadoop.apache.org/docs/stable2/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.Path,
long, long)
On Fri, Mar 28, 2014 at 7:33 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
--
Harsh J
Re: How to get locations of blocks programmatically?
Posted by Wangda Tan <wh...@gmail.com>.
Hi Libo,
DFSClient.getBlockLocations, is this what you want?
Regards,
Wangda Tan
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>
Re: How to get locations of blocks programmatically?
Posted by Stanley Shi <ss...@gopivotal.com>.
FileSystem.getFileBlockLocations(...)
Regards,
*Stanley Shi,*
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>
Re: How to get locations of blocks programmatically?
Posted by Stanley Shi <ss...@gopivotal.com>.
FileSystem.getFileBlockLocations(...)
Regards,
*Stanley Shi,*
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>
Re: How to get locations of blocks programmatically?
Posted by Hardik Pandya <sm...@gmail.com>.
have you looked into FileSystem API this is hadoop v2.2.0
http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html
does not exist in
http://hadoop.apache.org/docs/r1.2.0/api/org/apache/hadoop/fs/FileSystem.html
org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/LocatedFileStatus.html>
> *listFiles
<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html#listFiles%28org.apache.hadoop.fs.Path,%20boolean%29>*
(Path<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/Path.html>
f,
boolean recursive)
List the statuses and block locations of the files in the given
path. org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/LocatedFileStatus.html>
> *listLocatedStatus
<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html#listLocatedStatus%28org.apache.hadoop.fs.Path%29>*
(Path<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/Path.html>
f)
List the statuses of the files/directories in the given path if
the path is a directory.
On Thu, Mar 27, 2014 at 10:03 PM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>
Re: How to get locations of blocks programmatically?
Posted by Harsh J <ha...@cloudera.com>.
Yes, use http://hadoop.apache.org/docs/stable2/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.Path,
long, long)
On Fri, Mar 28, 2014 at 7:33 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
--
Harsh J
Re: How to get locations of blocks programmatically?
Posted by Hardik Pandya <sm...@gmail.com>.
have you looked into FileSystem API this is hadoop v2.2.0
http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html
does not exist in
http://hadoop.apache.org/docs/r1.2.0/api/org/apache/hadoop/fs/FileSystem.html
org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/LocatedFileStatus.html>
> *listFiles
<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html#listFiles%28org.apache.hadoop.fs.Path,%20boolean%29>*
(Path<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/Path.html>
f,
boolean recursive)
List the statuses and block locations of the files in the given
path. org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/LocatedFileStatus.html>
> *listLocatedStatus
<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html#listLocatedStatus%28org.apache.hadoop.fs.Path%29>*
(Path<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/Path.html>
f)
List the statuses of the files/directories in the given path if
the path is a directory.
On Thu, Mar 27, 2014 at 10:03 PM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>
Re: How to get locations of blocks programmatically?
Posted by Harsh J <ha...@cloudera.com>.
Yes, use http://hadoop.apache.org/docs/stable2/api/org/apache/hadoop/fs/FileSystem.html#getFileBlockLocations(org.apache.hadoop.fs.Path,
long, long)
On Fri, Mar 28, 2014 at 7:33 AM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
--
Harsh J
Re: How to get locations of blocks programmatically?
Posted by Hardik Pandya <sm...@gmail.com>.
have you looked into FileSystem API this is hadoop v2.2.0
http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html
does not exist in
http://hadoop.apache.org/docs/r1.2.0/api/org/apache/hadoop/fs/FileSystem.html
org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/LocatedFileStatus.html>
> *listFiles
<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html#listFiles%28org.apache.hadoop.fs.Path,%20boolean%29>*
(Path<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/Path.html>
f,
boolean recursive)
List the statuses and block locations of the files in the given
path. org.apache.hadoop.fs.RemoteIterator<LocatedFileStatus<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/LocatedFileStatus.html>
> *listLocatedStatus
<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html#listLocatedStatus%28org.apache.hadoop.fs.Path%29>*
(Path<http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/Path.html>
f)
List the statuses of the files/directories in the given path if
the path is a directory.
On Thu, Mar 27, 2014 at 10:03 PM, Libo Yu <yu...@hotmail.com> wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the block locations for a given path
> programmatically?
> Thanks,
>
> Libo
>