You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Pavan Kulkarni <pa...@gmail.com> on 2011/11/19 05:39:07 UTC
Doubt regarding finding Databaode --> Block map.
Hi all,
I am working on a class project which involves Development in HADOOP.
To implement a certain part I need the Datanode -> Block's map.(i.e. a
mapping which shows the list of blocks on a given datanode)
But I am unable to find this mapping in any of the classes. Any help would
be highly appreciated.
Thanking You,
--
With Regards
Pavan Kulkarni
Re: Doubt regarding finding Databaode --> Block map.
Posted by Harsh J <ha...@cloudera.com>.
Pavan,
Moving to hdfs-dev@hadoop.apache.org (bcc'd original destination).
Please use appropriate lists for queries to get the best responses.
Also helps stating what version/branch of Hadoop you're attempting to work on.
On Sat, Nov 19, 2011 at 10:09 AM, Pavan Kulkarni
<pa...@gmail.com> wrote:
> Hi all,
>
> I am working on a class project which involves Development in HADOOP.
> To implement a certain part I need the Datanode -> Block's map.(i.e. a
> mapping which shows the list of blocks on a given datanode)
> But I am unable to find this mapping in any of the classes. Any help would
> be highly appreciated.
What you're looking for should ideally be found at a NameNode.
FSNamesystem there would use a BlockManager, which uses a
DatanodeManager maintains a list of DataNodeDescriptors, and each of
those contain a BlockInfo 'list' structure, which carry finalized
blocks.
While I'm not sure what you're attempting to do or how you're
attempting to do it, if you have access to the live FSNamesystem
object, you may get a complete iterator for a node via:
namesystem.getBlockManager().getDatanodeManager().getDatanode(dataNodeID).getBlockIterator()
And then iterate upon that for each BlockInfo object, which has all
the per-block metadata you need.
Where dataNodeID is a DatanodeID identifier (You can alternatively use
the DatanodeManager#getDatanodeByHost(String host) call).
Happy hacking :)
(Please feel free to correct me if there's a more optimal, or direct
way to get what he wants.)
--
Harsh J
Re: Doubt regarding finding Databaode --> Block map.
Posted by Harsh J <ha...@cloudera.com>.
Pavan,
Moving to hdfs-dev@hadoop.apache.org (bcc'd original destination).
Please use appropriate lists for queries to get the best responses.
Also helps stating what version/branch of Hadoop you're attempting to work on.
On Sat, Nov 19, 2011 at 10:09 AM, Pavan Kulkarni
<pa...@gmail.com> wrote:
> Hi all,
>
> I am working on a class project which involves Development in HADOOP.
> To implement a certain part I need the Datanode -> Block's map.(i.e. a
> mapping which shows the list of blocks on a given datanode)
> But I am unable to find this mapping in any of the classes. Any help would
> be highly appreciated.
What you're looking for should ideally be found at a NameNode.
FSNamesystem there would use a BlockManager, which uses a
DatanodeManager maintains a list of DataNodeDescriptors, and each of
those contain a BlockInfo 'list' structure, which carry finalized
blocks.
While I'm not sure what you're attempting to do or how you're
attempting to do it, if you have access to the live FSNamesystem
object, you may get a complete iterator for a node via:
namesystem.getBlockManager().getDatanodeManager().getDatanode(dataNodeID).getBlockIterator()
And then iterate upon that for each BlockInfo object, which has all
the per-block metadata you need.
Where dataNodeID is a DatanodeID identifier (You can alternatively use
the DatanodeManager#getDatanodeByHost(String host) call).
Happy hacking :)
(Please feel free to correct me if there's a more optimal, or direct
way to get what he wants.)
--
Harsh J
Re: Doubt regarding finding Databaode --> Block map.
Posted by kartheek muthyala <ka...@gmail.com>.
There is a file called BlocksMap.java which does the same. You can also
refer to FSNamesystem.java which does the book keeping activity like
keeping track of several tables including machine->blocklist
~Kartheek
On Sat, Nov 19, 2011 at 10:09 AM, Pavan Kulkarni <pa...@gmail.com>wrote:
> Hi all,
>
> I am working on a class project which involves Development in HADOOP.
> To implement a certain part I need the Datanode -> Block's map.(i.e. a
> mapping which shows the list of blocks on a given datanode)
> But I am unable to find this mapping in any of the classes. Any help would
> be highly appreciated.
>
> Thanking You,
>
> --
> With Regards
> Pavan Kulkarni
>