You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by jr <jo...@io-consulting.net> on 2010/04/01 12:08:34 UTC
Re: Elephant Bird released
Hi Dmitriy,
does this require protobuf 2.3? I'm trying to build it on fedora and it
fails, i think it's because only 2.2 is available on fedora.
Best regards,
Johannes
Am Montag, den 29.03.2010, 14:51 -0700 schrieb Dmitriy Ryaboy:
> Hi folks,
> We (but mostly Kevin Weil) just open-sourced some of the code we use at
> Twitter to make working with Hadoop and Pig easier. Most of what is
> currently included in "Elephant Bird" deals with generating Input/Output
> formats for LZO-compressed protocol buffers, Pig LoadFuncs and StoreFuncs
> for the same; there are also some handy loaders for LZO-compressed stuff
> that is not probtobuf based.
>
> The project is on github: http://github.com/kevinweil/elephant-bird/
>
> Kevin presented on some of this at at HUG recently:
> http://www.slideshare.net/hadoopusergroup/twitter-protobufs-and-hadoop-hug-021709
>
> Feedback, bug reports, and patches are welcome! Hope you find this useful.
>
> -Dmitriy
Re: Elephant Bird released
Posted by jr <jo...@io-consulting.net>.
sorry for the post everyone!
johannes
Re: Elephant Bird released
Posted by jr <jo...@io-consulting.net>.
Hello Kevin,
I hope it's alright if i reply about this off the list since i don't
think it'd be helpful for now.
I'm trying to figure out what has to be compiled how, and first thing i
found is the package com.twitter.data.proto.BlockStorage
i can only find the javadoc for this and a few .java files importing
from that package.
is that being generated by protobuf? unfortunately i'm not familiar with
protobuf at all, so i'm not even sure how to generate that package at
all :)
What do i need to generate/get that?
Johannes
Am Donnerstag, den 01.04.2010, 07:37 -0700 schrieb Kevin Weil:
> Johannes, it does require protobuf 2.3. All of the inputformats, pig
> loaders, etc will themselves work on earlier versions of the protobuf
> library (we began on 2.2), but the protobuf codegen uses 2.3's new compiler
> plugin API. If you don't need that, you should be able to use 2.2 with a
> little hand editing.
>
> HTH,
> Kevin
>
> On Thu, Apr 1, 2010 at 3:08 AM, jr <jo...@io-consulting.net>wrote:
>
> > Hi Dmitriy,
> > does this require protobuf 2.3? I'm trying to build it on fedora and it
> > fails, i think it's because only 2.2 is available on fedora.
> > Best regards,
> > Johannes
> >
> > Am Montag, den 29.03.2010, 14:51 -0700 schrieb Dmitriy Ryaboy:
> > > Hi folks,
> > > We (but mostly Kevin Weil) just open-sourced some of the code we use at
> > > Twitter to make working with Hadoop and Pig easier. Most of what is
> > > currently included in "Elephant Bird" deals with generating Input/Output
> > > formats for LZO-compressed protocol buffers, Pig LoadFuncs and StoreFuncs
> > > for the same; there are also some handy loaders for LZO-compressed stuff
> > > that is not probtobuf based.
> > >
> > > The project is on github: http://github.com/kevinweil/elephant-bird/
> > >
> > > Kevin presented on some of this at at HUG recently:
> > >
> > http://www.slideshare.net/hadoopusergroup/twitter-protobufs-and-hadoop-hug-021709
> > >
> > > Feedback, bug reports, and patches are welcome! Hope you find this
> > useful.
> > >
> > > -Dmitriy
> >
> >
Re: Elephant Bird released
Posted by Kevin Weil <ke...@gmail.com>.
Johannes,
If you want to commit a patch to the build file with a "no-protobuf" target,
please do and send me a github pull request. I bet you aren't the only one
who will want this.
Thanks,
Kevin
On Thu, Apr 1, 2010 at 7:58 AM, jr <jo...@io-consulting.net>wrote:
> Hello Kevin,
> thanks a lot, since i really only need the pig loaders i'll go for hand
> editing :)
> Johannes
>
> Am Donnerstag, den 01.04.2010, 07:37 -0700 schrieb Kevin Weil:
> > Johannes, it does require protobuf 2.3. All of the inputformats, pig
> > loaders, etc will themselves work on earlier versions of the protobuf
> > library (we began on 2.2), but the protobuf codegen uses 2.3's new
> compiler
> > plugin API. If you don't need that, you should be able to use 2.2 with a
> > little hand editing.
> >
> > HTH,
> > Kevin
> >
> > On Thu, Apr 1, 2010 at 3:08 AM, jr <johannes.russek@io-consulting.net
> >wrote:
> >
> > > Hi Dmitriy,
> > > does this require protobuf 2.3? I'm trying to build it on fedora and it
> > > fails, i think it's because only 2.2 is available on fedora.
> > > Best regards,
> > > Johannes
> > >
> > > Am Montag, den 29.03.2010, 14:51 -0700 schrieb Dmitriy Ryaboy:
> > > > Hi folks,
> > > > We (but mostly Kevin Weil) just open-sourced some of the code we use
> at
> > > > Twitter to make working with Hadoop and Pig easier. Most of what is
> > > > currently included in "Elephant Bird" deals with generating
> Input/Output
> > > > formats for LZO-compressed protocol buffers, Pig LoadFuncs and
> StoreFuncs
> > > > for the same; there are also some handy loaders for LZO-compressed
> stuff
> > > > that is not probtobuf based.
> > > >
> > > > The project is on github: http://github.com/kevinweil/elephant-bird/
> > > >
> > > > Kevin presented on some of this at at HUG recently:
> > > >
> > >
> http://www.slideshare.net/hadoopusergroup/twitter-protobufs-and-hadoop-hug-021709
> > > >
> > > > Feedback, bug reports, and patches are welcome! Hope you find this
> > > useful.
> > > >
> > > > -Dmitriy
> > >
> > >
>
>
Re: Elephant Bird released
Posted by jr <jo...@io-consulting.net>.
Hello Kevin,
thanks a lot, since i really only need the pig loaders i'll go for hand
editing :)
Johannes
Am Donnerstag, den 01.04.2010, 07:37 -0700 schrieb Kevin Weil:
> Johannes, it does require protobuf 2.3. All of the inputformats, pig
> loaders, etc will themselves work on earlier versions of the protobuf
> library (we began on 2.2), but the protobuf codegen uses 2.3's new compiler
> plugin API. If you don't need that, you should be able to use 2.2 with a
> little hand editing.
>
> HTH,
> Kevin
>
> On Thu, Apr 1, 2010 at 3:08 AM, jr <jo...@io-consulting.net>wrote:
>
> > Hi Dmitriy,
> > does this require protobuf 2.3? I'm trying to build it on fedora and it
> > fails, i think it's because only 2.2 is available on fedora.
> > Best regards,
> > Johannes
> >
> > Am Montag, den 29.03.2010, 14:51 -0700 schrieb Dmitriy Ryaboy:
> > > Hi folks,
> > > We (but mostly Kevin Weil) just open-sourced some of the code we use at
> > > Twitter to make working with Hadoop and Pig easier. Most of what is
> > > currently included in "Elephant Bird" deals with generating Input/Output
> > > formats for LZO-compressed protocol buffers, Pig LoadFuncs and StoreFuncs
> > > for the same; there are also some handy loaders for LZO-compressed stuff
> > > that is not probtobuf based.
> > >
> > > The project is on github: http://github.com/kevinweil/elephant-bird/
> > >
> > > Kevin presented on some of this at at HUG recently:
> > >
> > http://www.slideshare.net/hadoopusergroup/twitter-protobufs-and-hadoop-hug-021709
> > >
> > > Feedback, bug reports, and patches are welcome! Hope you find this
> > useful.
> > >
> > > -Dmitriy
> >
> >
Re: Elephant Bird released
Posted by Kevin Weil <ke...@gmail.com>.
Johannes, it does require protobuf 2.3. All of the inputformats, pig
loaders, etc will themselves work on earlier versions of the protobuf
library (we began on 2.2), but the protobuf codegen uses 2.3's new compiler
plugin API. If you don't need that, you should be able to use 2.2 with a
little hand editing.
HTH,
Kevin
On Thu, Apr 1, 2010 at 3:08 AM, jr <jo...@io-consulting.net>wrote:
> Hi Dmitriy,
> does this require protobuf 2.3? I'm trying to build it on fedora and it
> fails, i think it's because only 2.2 is available on fedora.
> Best regards,
> Johannes
>
> Am Montag, den 29.03.2010, 14:51 -0700 schrieb Dmitriy Ryaboy:
> > Hi folks,
> > We (but mostly Kevin Weil) just open-sourced some of the code we use at
> > Twitter to make working with Hadoop and Pig easier. Most of what is
> > currently included in "Elephant Bird" deals with generating Input/Output
> > formats for LZO-compressed protocol buffers, Pig LoadFuncs and StoreFuncs
> > for the same; there are also some handy loaders for LZO-compressed stuff
> > that is not probtobuf based.
> >
> > The project is on github: http://github.com/kevinweil/elephant-bird/
> >
> > Kevin presented on some of this at at HUG recently:
> >
> http://www.slideshare.net/hadoopusergroup/twitter-protobufs-and-hadoop-hug-021709
> >
> > Feedback, bug reports, and patches are welcome! Hope you find this
> useful.
> >
> > -Dmitriy
>
>