You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2011/12/06 22:24:11 UTC
HBASE-4120 Was: trip report for Hadoop In China
Modified subject to match future discussion.
Andy:
In the latest patch for HBASE-4120, HRegionServer.addToOnlineRegions() is
used to refresh region priority.
Do you think RegionCoprocessorHost should be enhanced with more hooks, such
as preAddToOnlineRegions() ?
Thanks
On Tue, Dec 6, 2011 at 12:50 PM, Andrew Purtell <ap...@yahoo.com> wrote:
> Also. I was going to start separate threads but probably there is not that
> much to discuss now or off-JIRA:
>
> - I spoke with Jia Liu about HBASE-4120. After 0.92 is out I intend to
> champion / mentor / co-develop 4120 and the follow on table allocation work
> and target 0.94 for it. I think the RPC QoS aspect is not too
> controversial. The allocation/reservation aspects I'd like to aim for a
> coprocessor or at least master plugin based integration so they won't
> impact stability for users who don't enable it. Unlike RPC QoS I suspect
> the changes needed to core can be minimized to coprocessor framework
> additions. Follow up in new JIRAs soon.
>
> - I also met with the author of "CCIndex". He is still interested in
> pursuing an upstreaming of this but may not have a lot of free time. The
> current implementation powers the data cube product at Taobao with half a
> million users IIRC, but the implementation is an invasive hack of the 0.20
> core code. I think the vast majority of it is an application layer concern
> that can be moved to a coprocessor implementation. The coprocessor
> framework will need some modification as well, specifically hooks for
> handling IOEs taken on HFile reads. Anyway I'm going to play around with
> this and follow up with a JIRA.
>
> Best regards,
>
> - Andy
>
>
> On Dec 7, 2011, at 4:00 AM, Andrew Purtell <ap...@apache.org> wrote:
>
> > I also attended day 1 of Hadoop in China and would add these notes:
> >
> > - Taobao has their own HA NameNode and is interested in contributing
> their changes upstream. I spoke with a platform dev lead. I told him do so
> now while HA is under active development rather than wait and miss an
> opportunity to participate/contribute. I need to follow up with him with a
> list of active HA jiras.
> >
> > - Mingjie Lai presented on HBase and ZooKeeper security. I'll ask him to
> upload his slides somewhere.
> >
> >
> > Best regards,
> >
> >
> > - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
> >
> >
> > ----- Original Message -----
> >> From: Todd Lipcon <to...@cloudera.com>
> >> To: dev@hbase.apache.org
> >> Cc:
> >> Sent: Wednesday, December 7, 2011 2:59 AM
> >> Subject: Re: trip report for Hadoop In China
> >>
> >> Ah, I clicked on "hadooper.cn" directly instead of the earlier link to
> >> the agenda. Oops :)
> >>
> >> Maybe you can put them in a public S3 bucket?
> >>
> >> -Todd
> >>
> >> On Tue, Dec 6, 2011 at 10:56 AM, Ted Yu <yu...@gmail.com> wrote:
> >>> From http://hic2011.hadooper.cn/dct/page/1 (the first two lines):
> >>> Slides would be posted after getting permission from presenters
> >>>
> >>> I can share my slides. My attempt at uploading ppt to slideshare
> failed and
> >>> I don't have PRO account.
> >>>
> >>> I think apache.org would filter attachment.
> >>>
> >>> Suggestion on where I can put my slides would be appreciated.
> >>>
> >>> On Tue, Dec 6, 2011 at 10:39 AM, Todd Lipcon <to...@cloudera.com>
> >> wrote:
> >>>
> >>>> Hey Ted,
> >>>>
> >>>> Thanks for the trip report. Interesting stuff.
> >>>>
> >>>> For those of us who can't read mandarin, can you link directly to
> >> any
> >>>> english slide decks? The hadooper.cn link seems to go to an empty
> wiki
> >>>> template page or something, if I understand Google Translate correctly
> >>>> :)
> >>>>
> >>>> -Todd
> >>>>
> >>>> On Tue, Dec 6, 2011 at 10:36 AM, Ted Yu <yu...@gmail.com>
> >> wrote:
> >>>>> Hi,
> >>>>> There were around 1000 attendees for the first day of HiC.
> >>>>> The enthusiasm for cloud computing, hadoop in particular, was
> >>>> overwhelming.
> >>>>>
> >>>>> Doug Cutting's keynote was well received by attendees.
> >>>>> **Grzegorz's presentation about latest development @ Google,
> >> Pregel in
> >>>>> particular, was impressive.
> >>>>> Xiaodong Zhang's keynote about DOT, an analytical model, was
> >> interesting.
> >>>>>
> >>>>> I listened to several presentations. Here're a few:
> >>>>>
> >>>>> Zheng Shao talked about Puma3 architecture which his group
> >> developed at
> >>>>> Facebook
> >>>>> Jerry and Liyin talked about how Facebook messaging uses Hbase.
> >> Jerry
> >>>>> covered some best practices in using Hbase. Liyin talked about
> >> some
> >>>>> features developed by their team, including Hfile V2, new bloom
> >> filters,
> >>>> etc
> >>>>> Jinglong Liu from Baidu talked about how his team implemented
> >> compression
> >>>>> in hdfs (this is not RAID).
> >>>>>
> >>>>> I met with developers / managers from Taobao, Huawei, Renren
> >> (Chinese
> >>>>> version of Facebook), etc and shared experience using HBase.
> >>>>>
> >>>>> You can find the complete listing of talks here:
> >>>>> http://hic2011.hadooper.cn/dct/page/65580
> >>>>>
> >>>>> On my way back, I met Doug Cutting at the airport.
> >>>>> He endorsed the idea of promoting committers from China so that
> >> hadoop
> >>>>> community would have organic growth.
> >>>>>
> >>>>> I didn't take detailed notes. So please find the slides from
> >> hadooper.cn
> >>>>>
> >>>>> Thanks
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Todd Lipcon
> >>>> Software Engineer, Cloudera
> >>>>
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
>
Re: HBASE-4120 Was: trip report for Hadoop In China
Posted by Andrew Purtell <ap...@yahoo.com>.
Upcall
iPhone automiscorrect
On Dec 7, 2011, at 5:55 AM, Andrew Purtell <ap...@yahoo.com> wrote:
> I need to review this code.
>
> That said, why couldn't this be handled by a postOpen up all?
>
> Best regards,
>
> - Andy
>
>
> On Dec 7, 2011, at 5:24 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Modified subject to match future discussion.
>>
>> Andy:
>> In the latest patch for HBASE-4120, HRegionServer.addToOnlineRegions() is
>> used to refresh region priority.
>> Do you think RegionCoprocessorHost should be enhanced with more hooks, such
>> as preAddToOnlineRegions() ?
>>
>> Thanks
>>
>> On Tue, Dec 6, 2011 at 12:50 PM, Andrew Purtell <ap...@yahoo.com> wrote:
>>
>>> Also. I was going to start separate threads but probably there is not that
>>> much to discuss now or off-JIRA:
>>>
>>> - I spoke with Jia Liu about HBASE-4120. After 0.92 is out I intend to
>>> champion / mentor / co-develop 4120 and the follow on table allocation work
>>> and target 0.94 for it. I think the RPC QoS aspect is not too
>>> controversial. The allocation/reservation aspects I'd like to aim for a
>>> coprocessor or at least master plugin based integration so they won't
>>> impact stability for users who don't enable it. Unlike RPC QoS I suspect
>>> the changes needed to core can be minimized to coprocessor framework
>>> additions. Follow up in new JIRAs soon.
>>>
>>> - I also met with the author of "CCIndex". He is still interested in
>>> pursuing an upstreaming of this but may not have a lot of free time. The
>>> current implementation powers the data cube product at Taobao with half a
>>> million users IIRC, but the implementation is an invasive hack of the 0.20
>>> core code. I think the vast majority of it is an application layer concern
>>> that can be moved to a coprocessor implementation. The coprocessor
>>> framework will need some modification as well, specifically hooks for
>>> handling IOEs taken on HFile reads. Anyway I'm going to play around with
>>> this and follow up with a JIRA.
>>>
>>> Best regards,
>>>
>>> - Andy
>>>
>>>
>>> On Dec 7, 2011, at 4:00 AM, Andrew Purtell <ap...@apache.org> wrote:
>>>
>>>> I also attended day 1 of Hadoop in China and would add these notes:
>>>>
>>>> - Taobao has their own HA NameNode and is interested in contributing
>>> their changes upstream. I spoke with a platform dev lead. I told him do so
>>> now while HA is under active development rather than wait and miss an
>>> opportunity to participate/contribute. I need to follow up with him with a
>>> list of active HA jiras.
>>>>
>>>> - Mingjie Lai presented on HBase and ZooKeeper security. I'll ask him to
>>> upload his slides somewhere.
>>>>
>>>>
>>>> Best regards,
>>>>
>>>>
>>>> - Andy
>>>>
>>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: Todd Lipcon <to...@cloudera.com>
>>>>> To: dev@hbase.apache.org
>>>>> Cc:
>>>>> Sent: Wednesday, December 7, 2011 2:59 AM
>>>>> Subject: Re: trip report for Hadoop In China
>>>>>
>>>>> Ah, I clicked on "hadooper.cn" directly instead of the earlier link to
>>>>> the agenda. Oops :)
>>>>>
>>>>> Maybe you can put them in a public S3 bucket?
>>>>>
>>>>> -Todd
>>>>>
>>>>> On Tue, Dec 6, 2011 at 10:56 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>> From http://hic2011.hadooper.cn/dct/page/1 (the first two lines):
>>>>>> Slides would be posted after getting permission from presenters
>>>>>>
>>>>>> I can share my slides. My attempt at uploading ppt to slideshare
>>> failed and
>>>>>> I don't have PRO account.
>>>>>>
>>>>>> I think apache.org would filter attachment.
>>>>>>
>>>>>> Suggestion on where I can put my slides would be appreciated.
>>>>>>
>>>>>> On Tue, Dec 6, 2011 at 10:39 AM, Todd Lipcon <to...@cloudera.com>
>>>>> wrote:
>>>>>>
>>>>>>> Hey Ted,
>>>>>>>
>>>>>>> Thanks for the trip report. Interesting stuff.
>>>>>>>
>>>>>>> For those of us who can't read mandarin, can you link directly to
>>>>> any
>>>>>>> english slide decks? The hadooper.cn link seems to go to an empty
>>> wiki
>>>>>>> template page or something, if I understand Google Translate correctly
>>>>>>> :)
>>>>>>>
>>>>>>> -Todd
>>>>>>>
>>>>>>> On Tue, Dec 6, 2011 at 10:36 AM, Ted Yu <yu...@gmail.com>
>>>>> wrote:
>>>>>>>> Hi,
>>>>>>>> There were around 1000 attendees for the first day of HiC.
>>>>>>>> The enthusiasm for cloud computing, hadoop in particular, was
>>>>>>> overwhelming.
>>>>>>>>
>>>>>>>> Doug Cutting's keynote was well received by attendees.
>>>>>>>> **Grzegorz's presentation about latest development @ Google,
>>>>> Pregel in
>>>>>>>> particular, was impressive.
>>>>>>>> Xiaodong Zhang's keynote about DOT, an analytical model, was
>>>>> interesting.
>>>>>>>>
>>>>>>>> I listened to several presentations. Here're a few:
>>>>>>>>
>>>>>>>> Zheng Shao talked about Puma3 architecture which his group
>>>>> developed at
>>>>>>>> Facebook
>>>>>>>> Jerry and Liyin talked about how Facebook messaging uses Hbase.
>>>>> Jerry
>>>>>>>> covered some best practices in using Hbase. Liyin talked about
>>>>> some
>>>>>>>> features developed by their team, including Hfile V2, new bloom
>>>>> filters,
>>>>>>> etc
>>>>>>>> Jinglong Liu from Baidu talked about how his team implemented
>>>>> compression
>>>>>>>> in hdfs (this is not RAID).
>>>>>>>>
>>>>>>>> I met with developers / managers from Taobao, Huawei, Renren
>>>>> (Chinese
>>>>>>>> version of Facebook), etc and shared experience using HBase.
>>>>>>>>
>>>>>>>> You can find the complete listing of talks here:
>>>>>>>> http://hic2011.hadooper.cn/dct/page/65580
>>>>>>>>
>>>>>>>> On my way back, I met Doug Cutting at the airport.
>>>>>>>> He endorsed the idea of promoting committers from China so that
>>>>> hadoop
>>>>>>>> community would have organic growth.
>>>>>>>>
>>>>>>>> I didn't take detailed notes. So please find the slides from
>>>>> hadooper.cn
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Todd Lipcon
>>>>>>> Software Engineer, Cloudera
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Todd Lipcon
>>>>> Software Engineer, Cloudera
>>>>>
>>>
Re: HBASE-4120 Was: trip report for Hadoop In China
Posted by Andrew Purtell <ap...@yahoo.com>.
I need to review this code.
That said, why couldn't this be handled by a postOpen up all?
Best regards,
- Andy
On Dec 7, 2011, at 5:24 AM, Ted Yu <yu...@gmail.com> wrote:
> Modified subject to match future discussion.
>
> Andy:
> In the latest patch for HBASE-4120, HRegionServer.addToOnlineRegions() is
> used to refresh region priority.
> Do you think RegionCoprocessorHost should be enhanced with more hooks, such
> as preAddToOnlineRegions() ?
>
> Thanks
>
> On Tue, Dec 6, 2011 at 12:50 PM, Andrew Purtell <ap...@yahoo.com> wrote:
>
>> Also. I was going to start separate threads but probably there is not that
>> much to discuss now or off-JIRA:
>>
>> - I spoke with Jia Liu about HBASE-4120. After 0.92 is out I intend to
>> champion / mentor / co-develop 4120 and the follow on table allocation work
>> and target 0.94 for it. I think the RPC QoS aspect is not too
>> controversial. The allocation/reservation aspects I'd like to aim for a
>> coprocessor or at least master plugin based integration so they won't
>> impact stability for users who don't enable it. Unlike RPC QoS I suspect
>> the changes needed to core can be minimized to coprocessor framework
>> additions. Follow up in new JIRAs soon.
>>
>> - I also met with the author of "CCIndex". He is still interested in
>> pursuing an upstreaming of this but may not have a lot of free time. The
>> current implementation powers the data cube product at Taobao with half a
>> million users IIRC, but the implementation is an invasive hack of the 0.20
>> core code. I think the vast majority of it is an application layer concern
>> that can be moved to a coprocessor implementation. The coprocessor
>> framework will need some modification as well, specifically hooks for
>> handling IOEs taken on HFile reads. Anyway I'm going to play around with
>> this and follow up with a JIRA.
>>
>> Best regards,
>>
>> - Andy
>>
>>
>> On Dec 7, 2011, at 4:00 AM, Andrew Purtell <ap...@apache.org> wrote:
>>
>>> I also attended day 1 of Hadoop in China and would add these notes:
>>>
>>> - Taobao has their own HA NameNode and is interested in contributing
>> their changes upstream. I spoke with a platform dev lead. I told him do so
>> now while HA is under active development rather than wait and miss an
>> opportunity to participate/contribute. I need to follow up with him with a
>> list of active HA jiras.
>>>
>>> - Mingjie Lai presented on HBase and ZooKeeper security. I'll ask him to
>> upload his slides somewhere.
>>>
>>>
>>> Best regards,
>>>
>>>
>>> - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>>
>>>
>>> ----- Original Message -----
>>>> From: Todd Lipcon <to...@cloudera.com>
>>>> To: dev@hbase.apache.org
>>>> Cc:
>>>> Sent: Wednesday, December 7, 2011 2:59 AM
>>>> Subject: Re: trip report for Hadoop In China
>>>>
>>>> Ah, I clicked on "hadooper.cn" directly instead of the earlier link to
>>>> the agenda. Oops :)
>>>>
>>>> Maybe you can put them in a public S3 bucket?
>>>>
>>>> -Todd
>>>>
>>>> On Tue, Dec 6, 2011 at 10:56 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>> From http://hic2011.hadooper.cn/dct/page/1 (the first two lines):
>>>>> Slides would be posted after getting permission from presenters
>>>>>
>>>>> I can share my slides. My attempt at uploading ppt to slideshare
>> failed and
>>>>> I don't have PRO account.
>>>>>
>>>>> I think apache.org would filter attachment.
>>>>>
>>>>> Suggestion on where I can put my slides would be appreciated.
>>>>>
>>>>> On Tue, Dec 6, 2011 at 10:39 AM, Todd Lipcon <to...@cloudera.com>
>>>> wrote:
>>>>>
>>>>>> Hey Ted,
>>>>>>
>>>>>> Thanks for the trip report. Interesting stuff.
>>>>>>
>>>>>> For those of us who can't read mandarin, can you link directly to
>>>> any
>>>>>> english slide decks? The hadooper.cn link seems to go to an empty
>> wiki
>>>>>> template page or something, if I understand Google Translate correctly
>>>>>> :)
>>>>>>
>>>>>> -Todd
>>>>>>
>>>>>> On Tue, Dec 6, 2011 at 10:36 AM, Ted Yu <yu...@gmail.com>
>>>> wrote:
>>>>>>> Hi,
>>>>>>> There were around 1000 attendees for the first day of HiC.
>>>>>>> The enthusiasm for cloud computing, hadoop in particular, was
>>>>>> overwhelming.
>>>>>>>
>>>>>>> Doug Cutting's keynote was well received by attendees.
>>>>>>> **Grzegorz's presentation about latest development @ Google,
>>>> Pregel in
>>>>>>> particular, was impressive.
>>>>>>> Xiaodong Zhang's keynote about DOT, an analytical model, was
>>>> interesting.
>>>>>>>
>>>>>>> I listened to several presentations. Here're a few:
>>>>>>>
>>>>>>> Zheng Shao talked about Puma3 architecture which his group
>>>> developed at
>>>>>>> Facebook
>>>>>>> Jerry and Liyin talked about how Facebook messaging uses Hbase.
>>>> Jerry
>>>>>>> covered some best practices in using Hbase. Liyin talked about
>>>> some
>>>>>>> features developed by their team, including Hfile V2, new bloom
>>>> filters,
>>>>>> etc
>>>>>>> Jinglong Liu from Baidu talked about how his team implemented
>>>> compression
>>>>>>> in hdfs (this is not RAID).
>>>>>>>
>>>>>>> I met with developers / managers from Taobao, Huawei, Renren
>>>> (Chinese
>>>>>>> version of Facebook), etc and shared experience using HBase.
>>>>>>>
>>>>>>> You can find the complete listing of talks here:
>>>>>>> http://hic2011.hadooper.cn/dct/page/65580
>>>>>>>
>>>>>>> On my way back, I met Doug Cutting at the airport.
>>>>>>> He endorsed the idea of promoting committers from China so that
>>>> hadoop
>>>>>>> community would have organic growth.
>>>>>>>
>>>>>>> I didn't take detailed notes. So please find the slides from
>>>> hadooper.cn
>>>>>>>
>>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Todd Lipcon
>>>>>> Software Engineer, Cloudera
>>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>
>>