You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Sam William <sa...@stumbleupon.com> on 2012/01/23 22:42:26 UTC
LZO support for Pig-0.9.1
Hi,
Im looking for a solution to load/store lzo compressed files for Pig-0.9.1 . Any pointers would be useful.
Thanks,
Sam William
sampd@stumbleupon.com
Re: LZO support for Pig-0.9.1
Posted by Sam William <sa...@stumbleupon.com>.
Dmitriy/ Raghu,
Thanks very much.
Sam
On Jan 23, 2012, at 10:29 PM, Dmitriy Ryaboy wrote:
> It's far from experimental, Twitter's been running EB on pig 0.8 and 0.9
> for a year now.
>
> But you do need the protocol buffers 2.3 installed to build. We might just
> check in the generated code for the required bits so you don't have to some
> time soon..
>
> Dmitriy
>
>
> On Mon, Jan 23, 2012 at 2:00 PM, Sam William <sa...@stumbleupon.com> wrote:
>
>> I did read about this. Seems like it was experimental even for 0.8 . I
>> wasnt sure of 0.9. Moreover , I couldn't build it without protocol
>> buffer support. The noproto target seems to have been removed.
>>
>> Sam
>> On Jan 23, 2012, at 1:49 PM, Prashant Kommireddi wrote:
>>
>>> Hi Sam,
>>>
>>> Elephantbird might be something you are looking for:
>>> https://github.com/kevinweil/elephant-bird
>>>
>>> Thanks,
>>> Prashant
>>>
>>> On Mon, Jan 23, 2012 at 1:42 PM, Sam William <sa...@stumbleupon.com>
>> wrote:
>>>
>>>> Hi,
>>>> Im looking for a solution to load/store lzo compressed files for
>>>> Pig-0.9.1 . Any pointers would be useful.
>>>>
>>>> Thanks,
>>>>
>>>> Sam William
>>>> sampd@stumbleupon.com
>>>>
>>>>
>>>>
>>>>
>>
>> Sam William
>> sampd@stumbleupon.com
>>
>>
>>
>>
Sam William
sampd@stumbleupon.com
Re: LZO support for Pig-0.9.1
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
It's far from experimental, Twitter's been running EB on pig 0.8 and 0.9
for a year now.
But you do need the protocol buffers 2.3 installed to build. We might just
check in the generated code for the required bits so you don't have to some
time soon..
Dmitriy
On Mon, Jan 23, 2012 at 2:00 PM, Sam William <sa...@stumbleupon.com> wrote:
> I did read about this. Seems like it was experimental even for 0.8 . I
> wasnt sure of 0.9. Moreover , I couldn't build it without protocol
> buffer support. The noproto target seems to have been removed.
>
> Sam
> On Jan 23, 2012, at 1:49 PM, Prashant Kommireddi wrote:
>
> > Hi Sam,
> >
> > Elephantbird might be something you are looking for:
> > https://github.com/kevinweil/elephant-bird
> >
> > Thanks,
> > Prashant
> >
> > On Mon, Jan 23, 2012 at 1:42 PM, Sam William <sa...@stumbleupon.com>
> wrote:
> >
> >> Hi,
> >> Im looking for a solution to load/store lzo compressed files for
> >> Pig-0.9.1 . Any pointers would be useful.
> >>
> >> Thanks,
> >>
> >> Sam William
> >> sampd@stumbleupon.com
> >>
> >>
> >>
> >>
>
> Sam William
> sampd@stumbleupon.com
>
>
>
>
Re: LZO support for Pig-0.9.1
Posted by Sam William <sa...@stumbleupon.com>.
I did read about this. Seems like it was experimental even for 0.8 . I wasnt sure of 0.9. Moreover , I couldn't build it without protocol buffer support. The noproto target seems to have been removed.
Sam
On Jan 23, 2012, at 1:49 PM, Prashant Kommireddi wrote:
> Hi Sam,
>
> Elephantbird might be something you are looking for:
> https://github.com/kevinweil/elephant-bird
>
> Thanks,
> Prashant
>
> On Mon, Jan 23, 2012 at 1:42 PM, Sam William <sa...@stumbleupon.com> wrote:
>
>> Hi,
>> Im looking for a solution to load/store lzo compressed files for
>> Pig-0.9.1 . Any pointers would be useful.
>>
>> Thanks,
>>
>> Sam William
>> sampd@stumbleupon.com
>>
>>
>>
>>
Sam William
sampd@stumbleupon.com
Re: LZO support for Pig-0.9.1
Posted by Prashant Kommireddi <pr...@gmail.com>.
Hi Sam,
Elephantbird might be something you are looking for:
https://github.com/kevinweil/elephant-bird
Thanks,
Prashant
On Mon, Jan 23, 2012 at 1:42 PM, Sam William <sa...@stumbleupon.com> wrote:
> Hi,
> Im looking for a solution to load/store lzo compressed files for
> Pig-0.9.1 . Any pointers would be useful.
>
> Thanks,
>
> Sam William
> sampd@stumbleupon.com
>
>
>
>
Re: LZO support for Pig-0.9.1
Posted by Raghu Angadi <an...@gmail.com>.
btw, for simple use cases, you can load/store lzo files using PigStorage().
While stor
Only disadvantage is that files are not splittable.
grunt> set output.compression.enabled true;
grunt> set output.compression.codec com.hadoop.compression.lzo.LzopCodec;
grunt> x = load 'x.txt' using PigStorage();
grunt> store x into 'xout';
grunt> fs -ls xout;
Found 1 items
-rw-r--r-- 3 raghu users 68 2012-01-24 00:29
/user/raghu/xout/part-m-00000.lzo
Raghu.
On Mon, Jan 23, 2012 at 1:42 PM, Sam William <sa...@stumbleupon.com> wrote:
> Hi,
> Im looking for a solution to load/store lzo compressed files for
> Pig-0.9.1 . Any pointers would be useful.
>
> Thanks,
>
> Sam William
> sampd@stumbleupon.com
>
>
>
>