You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Saptarshi Guha <sa...@gmail.com> on 2011/03/21 21:11:09 UTC
Using the ElephantBird LzoBaseRegexLoader
Hello,
I have some LZO files, which i
a) indexed via DistributedLzoIndexer to create index files
b) did not index, so just some LZO files in a directory.
Using both approaches, I tried creating a subclass LzoBaseRegexLoader
that returns a pattern.
Sadly, not a single line matched. This is not a problem of the regex
(checked it works with other strings),
i modified LzoBaseRegexLoader.java to print the strings coming in and
I'm getting binary e.g.
http://pastebin.com/wAveGzDy
I'm using Pig 0.8 and ElephantBird checked out from
https://github.com/gerritjvv/elephant-bird
Any suggestions?
Saptarshi
Re: Using the ElephantBird LzoBaseRegexLoader
Posted by Gerrit Jansen van Vuuren <ge...@googlemail.com>.
Hi,
Several things you can try:
1)Try using com.twitter.elephantbird.pig.load.LzoPigStorage() and print out
a few lines just to make sure you can read clear text from the lzo files.
2) You can use this in combination with pigs REGEX_EXTRACT(String
expression, String regex, int matchIndex) built int function
3) Have you tried LzoRegexLoader(String pattern)?
Cheers,
Gerrit
On Mon, Mar 21, 2011 at 9:11 PM, Saptarshi Guha <sa...@gmail.com>wrote:
> Hello,
>
> I have some LZO files, which i
>
> a) indexed via DistributedLzoIndexer to create index files
> b) did not index, so just some LZO files in a directory.
>
> Using both approaches, I tried creating a subclass LzoBaseRegexLoader
> that returns a pattern.
> Sadly, not a single line matched. This is not a problem of the regex
> (checked it works with other strings),
> i modified LzoBaseRegexLoader.java to print the strings coming in and
> I'm getting binary e.g.
>
> http://pastebin.com/wAveGzDy
>
> I'm using Pig 0.8 and ElephantBird checked out from
> https://github.com/gerritjvv/elephant-bird
>
> Any suggestions?
>
> Saptarshi
>
Re: Using the ElephantBird LzoBaseRegexLoader
Posted by Saptarshi Guha <sa...@gmail.com>.
Did I also say, thank you to both of you and everyone involved for elephantbird.
An extremely useful set of tools (like a wonderful christmas present)
Cheers
Saptarshi
On Mon, Mar 21, 2011 at 4:50 PM, Saptarshi Guha
<sa...@gmail.com> wrote:
> Hi Dmitriy and Gerrit,
>
> I did the following,
>
> - confirmed that LzoPigStorage is indeed reading my lzo files (took
> this from gerrit's github)
> - confirmed that LzoBaseRegexLoader sublcass (mine) was getting
> strings (from pig-08 branch of Dmitriy's github)
>
> Both work. I was messing around (foolishly) with wrong extensions.
>
> Thanks
> Saptarshi
>
>
>
>
> On Mon, Mar 21, 2011 at 1:22 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>> Try the more up-to-date version
>> at https://github.com/dvryaboy/elephant-bird/tree/pig-08
>> please send me your class if it still fails, and the input (uncompressed)
>> data to reproduce the error.
>> D
>>
>> On Mon, Mar 21, 2011 at 1:11 PM, Saptarshi Guha <sa...@gmail.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> I have some LZO files, which i
>>>
>>> a) indexed via DistributedLzoIndexer to create index files
>>> b) did not index, so just some LZO files in a directory.
>>>
>>> Using both approaches, I tried creating a subclass LzoBaseRegexLoader
>>> that returns a pattern.
>>> Sadly, not a single line matched. This is not a problem of the regex
>>> (checked it works with other strings),
>>> i modified LzoBaseRegexLoader.java to print the strings coming in and
>>> I'm getting binary e.g.
>>>
>>> http://pastebin.com/wAveGzDy
>>>
>>> I'm using Pig 0.8 and ElephantBird checked out from
>>> https://github.com/gerritjvv/elephant-bird
>>>
>>> Any suggestions?
>>>
>>> Saptarshi
>>
>>
>
Re: Using the ElephantBird LzoBaseRegexLoader
Posted by Saptarshi Guha <sa...@gmail.com>.
Hi Dmitriy and Gerrit,
I did the following,
- confirmed that LzoPigStorage is indeed reading my lzo files (took
this from gerrit's github)
- confirmed that LzoBaseRegexLoader sublcass (mine) was getting
strings (from pig-08 branch of Dmitriy's github)
Both work. I was messing around (foolishly) with wrong extensions.
Thanks
Saptarshi
On Mon, Mar 21, 2011 at 1:22 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Try the more up-to-date version
> at https://github.com/dvryaboy/elephant-bird/tree/pig-08
> please send me your class if it still fails, and the input (uncompressed)
> data to reproduce the error.
> D
>
> On Mon, Mar 21, 2011 at 1:11 PM, Saptarshi Guha <sa...@gmail.com>
> wrote:
>>
>> Hello,
>>
>> I have some LZO files, which i
>>
>> a) indexed via DistributedLzoIndexer to create index files
>> b) did not index, so just some LZO files in a directory.
>>
>> Using both approaches, I tried creating a subclass LzoBaseRegexLoader
>> that returns a pattern.
>> Sadly, not a single line matched. This is not a problem of the regex
>> (checked it works with other strings),
>> i modified LzoBaseRegexLoader.java to print the strings coming in and
>> I'm getting binary e.g.
>>
>> http://pastebin.com/wAveGzDy
>>
>> I'm using Pig 0.8 and ElephantBird checked out from
>> https://github.com/gerritjvv/elephant-bird
>>
>> Any suggestions?
>>
>> Saptarshi
>
>
Re: Using the ElephantBird LzoBaseRegexLoader
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Try the more up-to-date version at
https://github.com/dvryaboy/elephant-bird/tree/pig-08
please send me your class if it still fails, and the input (uncompressed)
data to reproduce the error.
D
On Mon, Mar 21, 2011 at 1:11 PM, Saptarshi Guha <sa...@gmail.com>wrote:
> Hello,
>
> I have some LZO files, which i
>
> a) indexed via DistributedLzoIndexer to create index files
> b) did not index, so just some LZO files in a directory.
>
> Using both approaches, I tried creating a subclass LzoBaseRegexLoader
> that returns a pattern.
> Sadly, not a single line matched. This is not a problem of the regex
> (checked it works with other strings),
> i modified LzoBaseRegexLoader.java to print the strings coming in and
> I'm getting binary e.g.
>
> http://pastebin.com/wAveGzDy
>
> I'm using Pig 0.8 and ElephantBird checked out from
> https://github.com/gerritjvv/elephant-bird
>
> Any suggestions?
>
> Saptarshi
>