You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2011/12/06 19:36:20 UTC
trip report for Hadoop In China
Hi,
There were around 1000 attendees for the first day of HiC.
The enthusiasm for cloud computing, hadoop in particular, was overwhelming.
Doug Cutting's keynote was well received by attendees.
**Grzegorz's presentation about latest development @ Google, Pregel in
particular, was impressive.
Xiaodong Zhang's keynote about DOT, an analytical model, was interesting.
I listened to several presentations. Here're a few:
Zheng Shao talked about Puma3 architecture which his group developed at
Facebook
Jerry and Liyin talked about how Facebook messaging uses Hbase. Jerry
covered some best practices in using Hbase. Liyin talked about some
features developed by their team, including Hfile V2, new bloom filters, etc
Jinglong Liu from Baidu talked about how his team implemented compression
in hdfs (this is not RAID).
I met with developers / managers from Taobao, Huawei, Renren (Chinese
version of Facebook), etc and shared experience using HBase.
You can find the complete listing of talks here:
http://hic2011.hadooper.cn/dct/page/65580
On my way back, I met Doug Cutting at the airport.
He endorsed the idea of promoting committers from China so that hadoop
community would have organic growth.
I didn't take detailed notes. So please find the slides from hadooper.cn
Thanks
Re: trip report for Hadoop In China
Posted by Eugene Koontz <ek...@hiro-tan.org>.
On 12/6/11 12:00 PM, Andrew Purtell wrote:
> - Mingjie Lai presented on HBase and ZooKeeper security. I'll ask him to upload his slides somewhere.
>
>
>
That's great to hear. Related to that, I hope people will have a look
(and try out) HBase/Zookeeper Authentication by following the proposed
documentation available here:
https://issues.apache.org/jira/browse/HBASE-4960
-Eugene
Re: trip report for Hadoop In China
Posted by Andrew Purtell <ap...@yahoo.com>.
Also. I was going to start separate threads but probably there is not that much to discuss now or off-JIRA:
- I spoke with Jia Liu about HBASE-4120. After 0.92 is out I intend to champion / mentor / co-develop 4120 and the follow on table allocation work and target 0.94 for it. I think the RPC QoS aspect is not too controversial. The allocation/reservation aspects I'd like to aim for a coprocessor or at least master plugin based integration so they won't impact stability for users who don't enable it. Unlike RPC QoS I suspect the changes needed to core can be minimized to coprocessor framework additions. Follow up in new JIRAs soon.
- I also met with the author of "CCIndex". He is still interested in pursuing an upstreaming of this but may not have a lot of free time. The current implementation powers the data cube product at Taobao with half a million users IIRC, but the implementation is an invasive hack of the 0.20 core code. I think the vast majority of it is an application layer concern that can be moved to a coprocessor implementation. The coprocessor framework will need some modification as well, specifically hooks for handling IOEs taken on HFile reads. Anyway I'm going to play around with this and follow up with a JIRA.
Best regards,
- Andy
On Dec 7, 2011, at 4:00 AM, Andrew Purtell <ap...@apache.org> wrote:
> I also attended day 1 of Hadoop in China and would add these notes:
>
> - Taobao has their own HA NameNode and is interested in contributing their changes upstream. I spoke with a platform dev lead. I told him do so now while HA is under active development rather than wait and miss an opportunity to participate/contribute. I need to follow up with him with a list of active HA jiras.
>
> - Mingjie Lai presented on HBase and ZooKeeper security. I'll ask him to upload his slides somewhere.
>
>
> Best regards,
>
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
>
>
> ----- Original Message -----
>> From: Todd Lipcon <to...@cloudera.com>
>> To: dev@hbase.apache.org
>> Cc:
>> Sent: Wednesday, December 7, 2011 2:59 AM
>> Subject: Re: trip report for Hadoop In China
>>
>> Ah, I clicked on "hadooper.cn" directly instead of the earlier link to
>> the agenda. Oops :)
>>
>> Maybe you can put them in a public S3 bucket?
>>
>> -Todd
>>
>> On Tue, Dec 6, 2011 at 10:56 AM, Ted Yu <yu...@gmail.com> wrote:
>>> From http://hic2011.hadooper.cn/dct/page/1 (the first two lines):
>>> Slides would be posted after getting permission from presenters
>>>
>>> I can share my slides. My attempt at uploading ppt to slideshare failed and
>>> I don't have PRO account.
>>>
>>> I think apache.org would filter attachment.
>>>
>>> Suggestion on where I can put my slides would be appreciated.
>>>
>>> On Tue, Dec 6, 2011 at 10:39 AM, Todd Lipcon <to...@cloudera.com>
>> wrote:
>>>
>>>> Hey Ted,
>>>>
>>>> Thanks for the trip report. Interesting stuff.
>>>>
>>>> For those of us who can't read mandarin, can you link directly to
>> any
>>>> english slide decks? The hadooper.cn link seems to go to an empty wiki
>>>> template page or something, if I understand Google Translate correctly
>>>> :)
>>>>
>>>> -Todd
>>>>
>>>> On Tue, Dec 6, 2011 at 10:36 AM, Ted Yu <yu...@gmail.com>
>> wrote:
>>>>> Hi,
>>>>> There were around 1000 attendees for the first day of HiC.
>>>>> The enthusiasm for cloud computing, hadoop in particular, was
>>>> overwhelming.
>>>>>
>>>>> Doug Cutting's keynote was well received by attendees.
>>>>> **Grzegorz's presentation about latest development @ Google,
>> Pregel in
>>>>> particular, was impressive.
>>>>> Xiaodong Zhang's keynote about DOT, an analytical model, was
>> interesting.
>>>>>
>>>>> I listened to several presentations. Here're a few:
>>>>>
>>>>> Zheng Shao talked about Puma3 architecture which his group
>> developed at
>>>>> Facebook
>>>>> Jerry and Liyin talked about how Facebook messaging uses Hbase.
>> Jerry
>>>>> covered some best practices in using Hbase. Liyin talked about
>> some
>>>>> features developed by their team, including Hfile V2, new bloom
>> filters,
>>>> etc
>>>>> Jinglong Liu from Baidu talked about how his team implemented
>> compression
>>>>> in hdfs (this is not RAID).
>>>>>
>>>>> I met with developers / managers from Taobao, Huawei, Renren
>> (Chinese
>>>>> version of Facebook), etc and shared experience using HBase.
>>>>>
>>>>> You can find the complete listing of talks here:
>>>>> http://hic2011.hadooper.cn/dct/page/65580
>>>>>
>>>>> On my way back, I met Doug Cutting at the airport.
>>>>> He endorsed the idea of promoting committers from China so that
>> hadoop
>>>>> community would have organic growth.
>>>>>
>>>>> I didn't take detailed notes. So please find the slides from
>> hadooper.cn
>>>>>
>>>>> Thanks
>>>>
>>>>
>>>>
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
Re: trip report for Hadoop In China
Posted by Andrew Purtell <ap...@apache.org>.
I also attended day 1 of Hadoop in China and would add these notes:
- Taobao has their own HA NameNode and is interested in contributing their changes upstream. I spoke with a platform dev lead. I told him do so now while HA is under active development rather than wait and miss an opportunity to participate/contribute. I need to follow up with him with a list of active HA jiras.
- Mingjie Lai presented on HBase and ZooKeeper security. I'll ask him to upload his slides somewhere.
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
----- Original Message -----
> From: Todd Lipcon <to...@cloudera.com>
> To: dev@hbase.apache.org
> Cc:
> Sent: Wednesday, December 7, 2011 2:59 AM
> Subject: Re: trip report for Hadoop In China
>
> Ah, I clicked on "hadooper.cn" directly instead of the earlier link to
> the agenda. Oops :)
>
> Maybe you can put them in a public S3 bucket?
>
> -Todd
>
> On Tue, Dec 6, 2011 at 10:56 AM, Ted Yu <yu...@gmail.com> wrote:
>> From http://hic2011.hadooper.cn/dct/page/1 (the first two lines):
>> Slides would be posted after getting permission from presenters
>>
>> I can share my slides. My attempt at uploading ppt to slideshare failed and
>> I don't have PRO account.
>>
>> I think apache.org would filter attachment.
>>
>> Suggestion on where I can put my slides would be appreciated.
>>
>> On Tue, Dec 6, 2011 at 10:39 AM, Todd Lipcon <to...@cloudera.com>
> wrote:
>>
>>> Hey Ted,
>>>
>>> Thanks for the trip report. Interesting stuff.
>>>
>>> For those of us who can't read mandarin, can you link directly to
> any
>>> english slide decks? The hadooper.cn link seems to go to an empty wiki
>>> template page or something, if I understand Google Translate correctly
>>> :)
>>>
>>> -Todd
>>>
>>> On Tue, Dec 6, 2011 at 10:36 AM, Ted Yu <yu...@gmail.com>
> wrote:
>>> > Hi,
>>> > There were around 1000 attendees for the first day of HiC.
>>> > The enthusiasm for cloud computing, hadoop in particular, was
>>> overwhelming.
>>> >
>>> > Doug Cutting's keynote was well received by attendees.
>>> > **Grzegorz's presentation about latest development @ Google,
> Pregel in
>>> > particular, was impressive.
>>> > Xiaodong Zhang's keynote about DOT, an analytical model, was
> interesting.
>>> >
>>> > I listened to several presentations. Here're a few:
>>> >
>>> > Zheng Shao talked about Puma3 architecture which his group
> developed at
>>> > Facebook
>>> > Jerry and Liyin talked about how Facebook messaging uses Hbase.
> Jerry
>>> > covered some best practices in using Hbase. Liyin talked about
> some
>>> > features developed by their team, including Hfile V2, new bloom
> filters,
>>> etc
>>> > Jinglong Liu from Baidu talked about how his team implemented
> compression
>>> > in hdfs (this is not RAID).
>>> >
>>> > I met with developers / managers from Taobao, Huawei, Renren
> (Chinese
>>> > version of Facebook), etc and shared experience using HBase.
>>> >
>>> > You can find the complete listing of talks here:
>>> > http://hic2011.hadooper.cn/dct/page/65580
>>> >
>>> > On my way back, I met Doug Cutting at the airport.
>>> > He endorsed the idea of promoting committers from China so that
> hadoop
>>> > community would have organic growth.
>>> >
>>> > I didn't take detailed notes. So please find the slides from
> hadooper.cn
>>> >
>>> > Thanks
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: trip report for Hadoop In China
Posted by Liujia <li...@yahoo.cn>.
Hi :All slides of Hic2011 are available here:
http://hic2011.hadooper.cn/dct/page/65591
Best regards
Liu Jia
--- 11年12月7日,周三, Todd Lipcon <to...@cloudera.com> 写道:
发件人: Todd Lipcon <to...@cloudera.com>
主题: Re: trip report for Hadoop In China
收件人: dev@hbase.apache.org
日期: 2011年12月7日,周三,上午2:59
Ah, I clicked on "hadooper.cn" directly instead of the earlier link to
the agenda. Oops :)
Maybe you can put them in a public S3 bucket?
-Todd
On Tue, Dec 6, 2011 at 10:56 AM, Ted Yu <yu...@gmail.com> wrote:
> From http://hic2011.hadooper.cn/dct/page/1 (the first two lines):
> Slides would be posted after getting permission from presenters
>
> I can share my slides. My attempt at uploading ppt to slideshare failed and
> I don't have PRO account.
>
> I think apache.org would filter attachment.
>
> Suggestion on where I can put my slides would be appreciated.
>
> On Tue, Dec 6, 2011 at 10:39 AM, Todd Lipcon <to...@cloudera.com> wrote:
>
>> Hey Ted,
>>
>> Thanks for the trip report. Interesting stuff.
>>
>> For those of us who can't read mandarin, can you link directly to any
>> english slide decks? The hadooper.cn link seems to go to an empty wiki
>> template page or something, if I understand Google Translate correctly
>> :)
>>
>> -Todd
>>
>> On Tue, Dec 6, 2011 at 10:36 AM, Ted Yu <yu...@gmail.com> wrote:
>> > Hi,
>> > There were around 1000 attendees for the first day of HiC.
>> > The enthusiasm for cloud computing, hadoop in particular, was
>> overwhelming.
>> >
>> > Doug Cutting's keynote was well received by attendees.
>> > **Grzegorz's presentation about latest development @ Google, Pregel in
>> > particular, was impressive.
>> > Xiaodong Zhang's keynote about DOT, an analytical model, was interesting.
>> >
>> > I listened to several presentations. Here're a few:
>> >
>> > Zheng Shao talked about Puma3 architecture which his group developed at
>> > Facebook
>> > Jerry and Liyin talked about how Facebook messaging uses Hbase. Jerry
>> > covered some best practices in using Hbase. Liyin talked about some
>> > features developed by their team, including Hfile V2, new bloom filters,
>> etc
>> > Jinglong Liu from Baidu talked about how his team implemented compression
>> > in hdfs (this is not RAID).
>> >
>> > I met with developers / managers from Taobao, Huawei, Renren (Chinese
>> > version of Facebook), etc and shared experience using HBase.
>> >
>> > You can find the complete listing of talks here:
>> > http://hic2011.hadooper.cn/dct/page/65580
>> >
>> > On my way back, I met Doug Cutting at the airport.
>> > He endorsed the idea of promoting committers from China so that hadoop
>> > community would have organic growth.
>> >
>> > I didn't take detailed notes. So please find the slides from hadooper.cn
>> >
>> > Thanks
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
--
Todd Lipcon
Software Engineer, Cloudera
Re: trip report for Hadoop In China
Posted by Todd Lipcon <to...@cloudera.com>.
Ah, I clicked on "hadooper.cn" directly instead of the earlier link to
the agenda. Oops :)
Maybe you can put them in a public S3 bucket?
-Todd
On Tue, Dec 6, 2011 at 10:56 AM, Ted Yu <yu...@gmail.com> wrote:
> From http://hic2011.hadooper.cn/dct/page/1 (the first two lines):
> Slides would be posted after getting permission from presenters
>
> I can share my slides. My attempt at uploading ppt to slideshare failed and
> I don't have PRO account.
>
> I think apache.org would filter attachment.
>
> Suggestion on where I can put my slides would be appreciated.
>
> On Tue, Dec 6, 2011 at 10:39 AM, Todd Lipcon <to...@cloudera.com> wrote:
>
>> Hey Ted,
>>
>> Thanks for the trip report. Interesting stuff.
>>
>> For those of us who can't read mandarin, can you link directly to any
>> english slide decks? The hadooper.cn link seems to go to an empty wiki
>> template page or something, if I understand Google Translate correctly
>> :)
>>
>> -Todd
>>
>> On Tue, Dec 6, 2011 at 10:36 AM, Ted Yu <yu...@gmail.com> wrote:
>> > Hi,
>> > There were around 1000 attendees for the first day of HiC.
>> > The enthusiasm for cloud computing, hadoop in particular, was
>> overwhelming.
>> >
>> > Doug Cutting's keynote was well received by attendees.
>> > **Grzegorz's presentation about latest development @ Google, Pregel in
>> > particular, was impressive.
>> > Xiaodong Zhang's keynote about DOT, an analytical model, was interesting.
>> >
>> > I listened to several presentations. Here're a few:
>> >
>> > Zheng Shao talked about Puma3 architecture which his group developed at
>> > Facebook
>> > Jerry and Liyin talked about how Facebook messaging uses Hbase. Jerry
>> > covered some best practices in using Hbase. Liyin talked about some
>> > features developed by their team, including Hfile V2, new bloom filters,
>> etc
>> > Jinglong Liu from Baidu talked about how his team implemented compression
>> > in hdfs (this is not RAID).
>> >
>> > I met with developers / managers from Taobao, Huawei, Renren (Chinese
>> > version of Facebook), etc and shared experience using HBase.
>> >
>> > You can find the complete listing of talks here:
>> > http://hic2011.hadooper.cn/dct/page/65580
>> >
>> > On my way back, I met Doug Cutting at the airport.
>> > He endorsed the idea of promoting committers from China so that hadoop
>> > community would have organic growth.
>> >
>> > I didn't take detailed notes. So please find the slides from hadooper.cn
>> >
>> > Thanks
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
--
Todd Lipcon
Software Engineer, Cloudera
Re: trip report for Hadoop In China
Posted by Ted Yu <yu...@gmail.com>.
>From http://hic2011.hadooper.cn/dct/page/1 (the first two lines):
Slides would be posted after getting permission from presenters
I can share my slides. My attempt at uploading ppt to slideshare failed and
I don't have PRO account.
I think apache.org would filter attachment.
Suggestion on where I can put my slides would be appreciated.
On Tue, Dec 6, 2011 at 10:39 AM, Todd Lipcon <to...@cloudera.com> wrote:
> Hey Ted,
>
> Thanks for the trip report. Interesting stuff.
>
> For those of us who can't read mandarin, can you link directly to any
> english slide decks? The hadooper.cn link seems to go to an empty wiki
> template page or something, if I understand Google Translate correctly
> :)
>
> -Todd
>
> On Tue, Dec 6, 2011 at 10:36 AM, Ted Yu <yu...@gmail.com> wrote:
> > Hi,
> > There were around 1000 attendees for the first day of HiC.
> > The enthusiasm for cloud computing, hadoop in particular, was
> overwhelming.
> >
> > Doug Cutting's keynote was well received by attendees.
> > **Grzegorz's presentation about latest development @ Google, Pregel in
> > particular, was impressive.
> > Xiaodong Zhang's keynote about DOT, an analytical model, was interesting.
> >
> > I listened to several presentations. Here're a few:
> >
> > Zheng Shao talked about Puma3 architecture which his group developed at
> > Facebook
> > Jerry and Liyin talked about how Facebook messaging uses Hbase. Jerry
> > covered some best practices in using Hbase. Liyin talked about some
> > features developed by their team, including Hfile V2, new bloom filters,
> etc
> > Jinglong Liu from Baidu talked about how his team implemented compression
> > in hdfs (this is not RAID).
> >
> > I met with developers / managers from Taobao, Huawei, Renren (Chinese
> > version of Facebook), etc and shared experience using HBase.
> >
> > You can find the complete listing of talks here:
> > http://hic2011.hadooper.cn/dct/page/65580
> >
> > On my way back, I met Doug Cutting at the airport.
> > He endorsed the idea of promoting committers from China so that hadoop
> > community would have organic growth.
> >
> > I didn't take detailed notes. So please find the slides from hadooper.cn
> >
> > Thanks
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: trip report for Hadoop In China
Posted by Todd Lipcon <to...@cloudera.com>.
Hey Ted,
Thanks for the trip report. Interesting stuff.
For those of us who can't read mandarin, can you link directly to any
english slide decks? The hadooper.cn link seems to go to an empty wiki
template page or something, if I understand Google Translate correctly
:)
-Todd
On Tue, Dec 6, 2011 at 10:36 AM, Ted Yu <yu...@gmail.com> wrote:
> Hi,
> There were around 1000 attendees for the first day of HiC.
> The enthusiasm for cloud computing, hadoop in particular, was overwhelming.
>
> Doug Cutting's keynote was well received by attendees.
> **Grzegorz's presentation about latest development @ Google, Pregel in
> particular, was impressive.
> Xiaodong Zhang's keynote about DOT, an analytical model, was interesting.
>
> I listened to several presentations. Here're a few:
>
> Zheng Shao talked about Puma3 architecture which his group developed at
> Facebook
> Jerry and Liyin talked about how Facebook messaging uses Hbase. Jerry
> covered some best practices in using Hbase. Liyin talked about some
> features developed by their team, including Hfile V2, new bloom filters, etc
> Jinglong Liu from Baidu talked about how his team implemented compression
> in hdfs (this is not RAID).
>
> I met with developers / managers from Taobao, Huawei, Renren (Chinese
> version of Facebook), etc and shared experience using HBase.
>
> You can find the complete listing of talks here:
> http://hic2011.hadooper.cn/dct/page/65580
>
> On my way back, I met Doug Cutting at the airport.
> He endorsed the idea of promoting committers from China so that hadoop
> community would have organic growth.
>
> I didn't take detailed notes. So please find the slides from hadooper.cn
>
> Thanks
--
Todd Lipcon
Software Engineer, Cloudera