You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Gokulakannan M <go...@huawei.com> on 2011/01/04 06:55:01 UTC
RE: Query reg hbase and hadoop append
Hello,
I want to merge the append feature from hadoop 0.20-append to the
hadoop 0.20.1 release which I am using currently.
Can you please give the roadmap for append implementation in hadoop
0.20-append(JIRA issues or related details)? All I know is HDFS-200 only :(
Thanks and Regards,
Gokul
-----Original Message-----
From: Gokulakannan M [mailto:gokulm@huawei.com]
Sent: Wednesday, December 29, 2010 10:53 AM
To: user@hbase.apache.org
Cc: saint.ack@gmail.com
Subject: RE: Query reg hbase and hadoop append
Thank you St.Ack.
This helps a lot.. :)
-----Original Message-----
From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
Sent: Tuesday, December 28, 2010 11:02 PM
To: user@hbase.apache.org; gokulm@huawei.com
Subject: Re: Query reg hbase and hadoop append
We have been concentrating on delivering an hbase that works with
hadoop's branch-0.20-append branch and CDH3. We've not spent time on
proofing an hbase on top of hadoop 0.21. I myself haven't tried it in
a while. There is reflection code around the sync call which tries to
figure which version of Hadoop we are on top of. I'd imagine that
while it might work, it might not be as performant as when run against
hadoop 0.20 append since this is where we've been spending our time.
Here is our story on hadoop versions for the coming 0.90.0:
http://people.apache.org/~stack/hbase-0.90.0-candidate-2/docs/notsoquick.htm
l#hadoop
(Is this clear? Should we add anything?).
As to why we're in this state where we are NOT up against the latest
version of hadoop, the story is a little involved. I'll spare you the
torturous detail. In short, the next major HBase release was to run
against hadoop 0.21 but when we learned, belatedly, that its release
was delayed with no guarantees regards when it would drop and that it
was not going to be installed (or tested) by the 'big boys' -- Y!, FB,
etc. -- we lost all enthusiasm for hadoop 0.21. We figured no one
would run an hadoop 0.21 in production.
Our focus shifted to the append branch in 0.20 because this was a
relatively minor deviation from the pervasive hadoop 0.20. We could
envision folks installing the append branch to get an hbase
that-did-not-lose-data and running this hadoop in production because
it was close enough to hadoop 0.20 and at least one of the big teams
was running it -- i.e. fb -- or something very close to it. A bunch
of effort was sunk into making sure HBase worked on this branch and
that it was performant.
Cloudera's CDH3 since b2 has had an append. Its provenance is the
hadoop 0.20-append branch. Not hadoop 0.21.
Hope this helps,
St.Ack
On Tue, Dec 28, 2010 at 12:43 AM, Gokulakannan M <go...@huawei.com> wrote:
> hadoop 0.21.0 has been released with the stable hdfs append feature..
>
>
>
> Should we do anything in hbase(modify / apply some patch) to use this hdfs
> append feature or hbase takes care of this in-built??
>
>
>
> from web i've seen that this append feature(hdfs-265) is different from
what
> hbase expects(hdfs-200)..
>
>
>
> can anyone pls explain the background???
>
>
>
>
>
> Thanks,
>
> Gokul
>
>
>
>
>
>
RE: Query reg hbase and hadoop append
Posted by Gokulakannan M <go...@huawei.com>.
Hi St.Ack,
Correct. It is little tedious(the changes are huge) to merge all the
patches rather than running the branch.
I am going to run the 0.20-append branch only. Thanks.. :-)
-----Original Message-----
From: stack [mailto:saint.ack@gmail.com]
Sent: Tuesday, January 04, 2011 9:53 PM
To: user@hbase.apache.org; gokulm@huawei.com
Subject: Re: Query reg hbase and hadoop append
Here are the list of changes that go to make up 0.20-append:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-append/CHANG
ES.txt?revision=1044210&view=markup
My guess is that it'd be a bit of an undertaking adding in all the patches
from this branch. You don't want to just run the branch? Or there is also
Cloudera' CDH?
St.Ack
On Mon, Jan 3, 2011 at 9:55 PM, Gokulakannan M <go...@huawei.com> wrote:
> Hello,
>
>
>
> I want to merge the append feature from hadoop 0.20-append to the
> hadoop 0.20.1 release which I am using currently.
>
>
>
> Can you please give the roadmap for append implementation in
> hadoop 0.20-append(JIRA issues or related details)? All I know is
> HDFS-200 only :(
>
>
>
> Thanks and Regards,
>
> Gokul
>
>
>
>
>
> -----Original Message-----
> From: Gokulakannan M [mailto:gokulm@huawei.com]
> Sent: Wednesday, December 29, 2010 10:53 AM
> To: user@hbase.apache.org
> Cc: saint.ack@gmail.com
> Subject: RE: Query reg hbase and hadoop append
>
>
>
> Thank you St.Ack.
>
>
>
> This helps a lot.. :)
>
>
>
>
>
>
>
> -----Original Message-----
>
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> Stack
>
> Sent: Tuesday, December 28, 2010 11:02 PM
>
> To: user@hbase.apache.org; gokulm@huawei.com
>
> Subject: Re: Query reg hbase and hadoop append
>
>
>
>
>
>
>
> We have been concentrating on delivering an hbase that works with
>
>
>
> hadoop's branch-0.20-append branch and CDH3. We've not spent time on
>
>
>
> proofing an hbase on top of hadoop 0.21. I myself haven't tried it in
>
>
>
> a while. There is reflection code around the sync call which tries to
>
>
>
> figure which version of Hadoop we are on top of. I'd imagine that
>
>
>
> while it might work, it might not be as performant as when run against
>
>
>
> hadoop 0.20 append since this is where we've been spending our time.
>
>
>
>
>
>
>
> Here is our story on hadoop versions for the coming 0.90.0:
>
>
>
> http://people.apache.org/~stack/hbase-0.90.0-candidate-2/docs/notsoqui
> ck.htm
>
> l#hadoop
>
>
>
> (Is this clear? Should we add anything?).
>
>
>
>
>
>
>
> As to why we're in this state where we are NOT up against the latest
>
>
>
> version of hadoop, the story is a little involved. I'll spare you the
>
>
>
> torturous detail. In short, the next major HBase release was to run
>
>
>
> against hadoop 0.21 but when we learned, belatedly, that its release
>
>
>
> was delayed with no guarantees regards when it would drop and that it
>
>
>
> was not going to be installed (or tested) by the 'big boys' -- Y!, FB,
>
>
>
> etc. -- we lost all enthusiasm for hadoop 0.21. We figured no one
>
>
>
> would run an hadoop 0.21 in production.
>
>
>
>
>
>
>
> Our focus shifted to the append branch in 0.20 because this was a
>
>
>
> relatively minor deviation from the pervasive hadoop 0.20. We could
>
>
>
> envision folks installing the append branch to get an hbase
>
>
>
> that-did-not-lose-data and running this hadoop in production because
>
>
>
> it was close enough to hadoop 0.20 and at least one of the big teams
>
>
>
> was running it -- i.e. fb -- or something very close to it. A bunch
>
>
>
> of effort was sunk into making sure HBase worked on this branch and
>
>
>
> that it was performant.
>
>
>
>
>
>
>
> Cloudera's CDH3 since b2 has had an append. Its provenance is the
>
>
>
> hadoop 0.20-append branch. Not hadoop 0.21.
>
>
>
>
>
>
>
> Hope this helps,
>
>
>
> St.Ack
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Dec 28, 2010 at 12:43 AM, Gokulakannan M <go...@huawei.com>
wrote:
>
>
>
>> hadoop 0.21.0 has been released with the stable hdfs append feature..
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>> Should we do anything in hbase(modify / apply some patch) to use this
>> hdfs
>
>
>
>> append feature or hbase takes care of this in-built??
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>> from web i've seen that this append feature(hdfs-265) is different
>> from
>
> what
>
>
>
>> hbase expects(hdfs-200)..
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>> can anyone pls explain the background???
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>> Thanks,
>
>
>
>>
>
>
>
>> Gokul
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>
Re: Query reg hbase and hadoop append
Posted by stack <sa...@gmail.com>.
Here are the list of changes that go to make up 0.20-append:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-append/CHANGES.txt?revision=1044210&view=markup
My guess is that it'd be a bit of an undertaking adding in all the
patches from this branch. You don't want to just run the branch? Or
there is also Cloudera' CDH?
St.Ack
On Mon, Jan 3, 2011 at 9:55 PM, Gokulakannan M <go...@huawei.com> wrote:
> Hello,
>
>
>
> I want to merge the append feature from hadoop 0.20-append to the
> hadoop 0.20.1 release which I am using currently.
>
>
>
> Can you please give the roadmap for append implementation in hadoop
> 0.20-append(JIRA issues or related details)? All I know is HDFS-200 only :(
>
>
>
> Thanks and Regards,
>
> Gokul
>
>
>
>
>
> -----Original Message-----
> From: Gokulakannan M [mailto:gokulm@huawei.com]
> Sent: Wednesday, December 29, 2010 10:53 AM
> To: user@hbase.apache.org
> Cc: saint.ack@gmail.com
> Subject: RE: Query reg hbase and hadoop append
>
>
>
> Thank you St.Ack.
>
>
>
> This helps a lot.. :)
>
>
>
>
>
>
>
> -----Original Message-----
>
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
>
> Sent: Tuesday, December 28, 2010 11:02 PM
>
> To: user@hbase.apache.org; gokulm@huawei.com
>
> Subject: Re: Query reg hbase and hadoop append
>
>
>
>
>
>
>
> We have been concentrating on delivering an hbase that works with
>
>
>
> hadoop's branch-0.20-append branch and CDH3. We've not spent time on
>
>
>
> proofing an hbase on top of hadoop 0.21. I myself haven't tried it in
>
>
>
> a while. There is reflection code around the sync call which tries to
>
>
>
> figure which version of Hadoop we are on top of. I'd imagine that
>
>
>
> while it might work, it might not be as performant as when run against
>
>
>
> hadoop 0.20 append since this is where we've been spending our time.
>
>
>
>
>
>
>
> Here is our story on hadoop versions for the coming 0.90.0:
>
>
>
> http://people.apache.org/~stack/hbase-0.90.0-candidate-2/docs/notsoquick.htm
>
> l#hadoop
>
>
>
> (Is this clear? Should we add anything?).
>
>
>
>
>
>
>
> As to why we're in this state where we are NOT up against the latest
>
>
>
> version of hadoop, the story is a little involved. I'll spare you the
>
>
>
> torturous detail. In short, the next major HBase release was to run
>
>
>
> against hadoop 0.21 but when we learned, belatedly, that its release
>
>
>
> was delayed with no guarantees regards when it would drop and that it
>
>
>
> was not going to be installed (or tested) by the 'big boys' -- Y!, FB,
>
>
>
> etc. -- we lost all enthusiasm for hadoop 0.21. We figured no one
>
>
>
> would run an hadoop 0.21 in production.
>
>
>
>
>
>
>
> Our focus shifted to the append branch in 0.20 because this was a
>
>
>
> relatively minor deviation from the pervasive hadoop 0.20. We could
>
>
>
> envision folks installing the append branch to get an hbase
>
>
>
> that-did-not-lose-data and running this hadoop in production because
>
>
>
> it was close enough to hadoop 0.20 and at least one of the big teams
>
>
>
> was running it -- i.e. fb -- or something very close to it. A bunch
>
>
>
> of effort was sunk into making sure HBase worked on this branch and
>
>
>
> that it was performant.
>
>
>
>
>
>
>
> Cloudera's CDH3 since b2 has had an append. Its provenance is the
>
>
>
> hadoop 0.20-append branch. Not hadoop 0.21.
>
>
>
>
>
>
>
> Hope this helps,
>
>
>
> St.Ack
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Tue, Dec 28, 2010 at 12:43 AM, Gokulakannan M <go...@huawei.com> wrote:
>
>
>
>> hadoop 0.21.0 has been released with the stable hdfs append feature..
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>> Should we do anything in hbase(modify / apply some patch) to use this hdfs
>
>
>
>> append feature or hbase takes care of this in-built??
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>> from web i've seen that this append feature(hdfs-265) is different from
>
> what
>
>
>
>> hbase expects(hdfs-200)..
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>> can anyone pls explain the background???
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>> Thanks,
>
>
>
>>
>
>
>
>> Gokul
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>>
>
>
>
>