You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Yu Li <ca...@gmail.com> on 2014/01/15 17:46:26 UTC
Is there a way to get a Block through block id?
Dear all,
As titled, I actually have two questions here:
1. In current releases like hadoop-2.2.0, is block id unique and able to
locate a Block in HDFS? I'm asking because I could see HDFS-4645 is trying
to resolve the uniqueness issue. However, from the code comment it seems
block id is expected to be unique
2. It seems there's no method to get a Block object through a block id.
However, there's some scenarios need such method, like if I use the
FavouredNode feature to create logical datanode group and planned to put
some data within a group, then I might need to periodically check whether
there's block somehow placed outside the group, and move it back. In such
scenario, I would need to first get block ids, then move. But to move them,
it seems we need a Block instance to initiate the ExtendedBlock object.
Any suggestion, or reference to existing JIRA would be highly appreciated,
and thanks in advance!
--
Best Regards,
Li Yu
Re: Is there a way to get a Block through block id?
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
You could look at the BlocksMap. That is where blocks should reside. It
depends on what you're trying to do.
cheers,
Colin
On Tue, Jan 21, 2014 at 10:00 PM, Yu Li <ca...@gmail.com> wrote:
> Hi Colin,
>
> Thanks for the reply. I guess you're referring to the below methods? If so,
> I'm afraid it can only get an empty block but not the real block I really
> want to move.
> =====================================
> public ExtendedBlock(final String poolId, final long blockId) {
> this(poolId, blockId, 0, 0);
> }
> public ExtendedBlock(final String poolId, final long blkid, final long
> len,
> final long genstamp) {
> this.poolId = poolId;
> block = new Block(blkid, len, genstamp);
> }
> =====================================
>
> I'm now working around this by adding an option to directly move the block
> during fsck (NmaenodeFsck). But still, if there's any method to directly
> get the Block instance by blockid through any api, it would be great to
> know. :-)
>
> On 21 January 2014 03:46, Colin McCabe <cm...@alumni.cmu.edu> wrote:
>
> > In order to uniquely identify a block in hadoop 2.2, you are going to
> need
> > both a block and a block pool ID. You can construct a Block object with
> > those two items.
> >
> > On Wed, Jan 15, 2014 at 8:46 AM, Yu Li <ca...@gmail.com> wrote:
> >
> > > Dear all,
> > >
> > > As titled, I actually have two questions here:
> > >
> > > 1. In current releases like hadoop-2.2.0, is block id unique and able
> to
> > > locate a Block in HDFS? I'm asking because I could see HDFS-4645 is
> > trying
> > > to resolve the uniqueness issue. However, from the code comment it
> seems
> > > block id is expected to be unique
> > >
> > >
> > It's expected to be unique within a block pool. You can get the block
> pool
> > ID you are using from your FSNamesystem object in 2.2.
> >
> > best,
> > Colin
> >
> > 2. It seems there's no method to get a Block object through a block id.
> > > However, there's some scenarios need such method, like if I use the
> > > FavouredNode feature to create logical datanode group and planned to
> put
> > > some data within a group, then I might need to periodically check
> whether
> > > there's block somehow placed outside the group, and move it back. In
> such
> > > scenario, I would need to first get block ids, then move. But to move
> > them,
> > > it seems we need a Block instance to initiate the ExtendedBlock object.
> > >
> > > Any suggestion, or reference to existing JIRA would be highly
> > appreciated,
> > > and thanks in advance!
> > >
> > > --
> > > Best Regards,
> > > Li Yu
> > >
> >
>
>
>
> --
> Best Regards,
> Li Yu
>
Re: Is there a way to get a Block through block id?
Posted by Yu Li <ca...@gmail.com>.
Hi Colin,
Thanks for the reply. I guess you're referring to the below methods? If so,
I'm afraid it can only get an empty block but not the real block I really
want to move.
=====================================
public ExtendedBlock(final String poolId, final long blockId) {
this(poolId, blockId, 0, 0);
}
public ExtendedBlock(final String poolId, final long blkid, final long
len,
final long genstamp) {
this.poolId = poolId;
block = new Block(blkid, len, genstamp);
}
=====================================
I'm now working around this by adding an option to directly move the block
during fsck (NmaenodeFsck). But still, if there's any method to directly
get the Block instance by blockid through any api, it would be great to
know. :-)
On 21 January 2014 03:46, Colin McCabe <cm...@alumni.cmu.edu> wrote:
> In order to uniquely identify a block in hadoop 2.2, you are going to need
> both a block and a block pool ID. You can construct a Block object with
> those two items.
>
> On Wed, Jan 15, 2014 at 8:46 AM, Yu Li <ca...@gmail.com> wrote:
>
> > Dear all,
> >
> > As titled, I actually have two questions here:
> >
> > 1. In current releases like hadoop-2.2.0, is block id unique and able to
> > locate a Block in HDFS? I'm asking because I could see HDFS-4645 is
> trying
> > to resolve the uniqueness issue. However, from the code comment it seems
> > block id is expected to be unique
> >
> >
> It's expected to be unique within a block pool. You can get the block pool
> ID you are using from your FSNamesystem object in 2.2.
>
> best,
> Colin
>
> 2. It seems there's no method to get a Block object through a block id.
> > However, there's some scenarios need such method, like if I use the
> > FavouredNode feature to create logical datanode group and planned to put
> > some data within a group, then I might need to periodically check whether
> > there's block somehow placed outside the group, and move it back. In such
> > scenario, I would need to first get block ids, then move. But to move
> them,
> > it seems we need a Block instance to initiate the ExtendedBlock object.
> >
> > Any suggestion, or reference to existing JIRA would be highly
> appreciated,
> > and thanks in advance!
> >
> > --
> > Best Regards,
> > Li Yu
> >
>
--
Best Regards,
Li Yu
Re: Is there a way to get a Block through block id?
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
In order to uniquely identify a block in hadoop 2.2, you are going to need
both a block and a block pool ID. You can construct a Block object with
those two items.
On Wed, Jan 15, 2014 at 8:46 AM, Yu Li <ca...@gmail.com> wrote:
> Dear all,
>
> As titled, I actually have two questions here:
>
> 1. In current releases like hadoop-2.2.0, is block id unique and able to
> locate a Block in HDFS? I'm asking because I could see HDFS-4645 is trying
> to resolve the uniqueness issue. However, from the code comment it seems
> block id is expected to be unique
>
>
It's expected to be unique within a block pool. You can get the block pool
ID you are using from your FSNamesystem object in 2.2.
best,
Colin
2. It seems there's no method to get a Block object through a block id.
> However, there's some scenarios need such method, like if I use the
> FavouredNode feature to create logical datanode group and planned to put
> some data within a group, then I might need to periodically check whether
> there's block somehow placed outside the group, and move it back. In such
> scenario, I would need to first get block ids, then move. But to move them,
> it seems we need a Block instance to initiate the ExtendedBlock object.
>
> Any suggestion, or reference to existing JIRA would be highly appreciated,
> and thanks in advance!
>
> --
> Best Regards,
> Li Yu
>