You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Benjamin Juhn <be...@gmail.com> on 2012/04/04 01:32:24 UTC
Re: deserializing nested protobufs
Thanks Dmitriy. Doesn't look like that class supports extensions. Am I missing something?
- Ben
On Mar 27, 2012, at 10:01 PM, Dmitriy Ryaboy wrote:
> I think you want ProtobufBytesToTuple
> (https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/piggybank/ProtobufBytesToTuple.java)
>
> This uses ProtobufToPig to do the actual conversion, and wraps it in a UDF.
>
> D
>
> On Tue, Mar 27, 2012 at 7:21 PM, Benjamin Juhn <be...@gmail.com> wrote:
>> Hey Jon,
>>
>> I'm not seeing anything in ProtobuftoPig that is specific to handling protobuf byte fields containing serialized protobuf.
>>
>> Do you have a Pig example?
>>
>> Thanks,
>> Ben
>>
>> On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:
>>
>>> Look up the util class ProtobuftoPig
>>>
>>> 2012/3/26 Benjamin Juhn <be...@gmail.com>
>>>
>>>> Thanks Jon,
>>>>
>>>> I'm using elephant-bird, and it allows me to load message A. I'm not sure
>>>> if it has pig bindings allowing me to deserialize message B from message A
>>>> though.
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
>>>>
>>>>> Ben,
>>>>>
>>>>> take a look at elephantbird https://github.com/kevinweil/elephant-bird
>>>>>
>>>>> Jon
>>>>>
>>>>> 2012/3/26 Benjamin Juhn <be...@gmail.com>
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'm using pig with protobuf and I have some byte fields containing
>>>>>> serialized protobuf data. Is it possible to handle this nested
>>>> serialized
>>>>>> data with pig?
>>>>>>
>>>>>> ex.
>>>>>> message A {
>>>>>> required bytes data = 1 // serialized message B
>>>>>> }
>>>>>> message B {
>>>>>> ...
>>>>>> }
>>>>>>
>>>>>> Thanks,
>>>>>> Ben
>>>>
>>>>
>>
Re: deserializing nested protobufs
Posted by Benjamin Juhn <be...@gmail.com>.
Looks like it's covered:
public ProtobufBytesToTuple(TypeRef<M> typeRef, ProtobufExtensionRegistry extensionRegistry) {
Thanks,
Ben
On Apr 3, 2012, at 4:41 PM, Raghu Angadi wrote:
> extension are not supported yet. there is a patch pending :
> https://github.com/kevinweil/elephant-bird/pull/143
>
> Can you check if that covers your use case?
>
> On Tue, Apr 3, 2012 at 4:32 PM, Benjamin Juhn <be...@gmail.com> wrote:
>
>> Thanks Dmitriy. Doesn't look like that class supports extensions. Am I
>> missing something?
>>
>> - Ben
>>
>> On Mar 27, 2012, at 10:01 PM, Dmitriy Ryaboy wrote:
>>
>>> I think you want ProtobufBytesToTuple
>>> (
>> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/piggybank/ProtobufBytesToTuple.java
>> )
>>>
>>> This uses ProtobufToPig to do the actual conversion, and wraps it in a
>> UDF.
>>>
>>> D
>>>
>>> On Tue, Mar 27, 2012 at 7:21 PM, Benjamin Juhn <be...@gmail.com>
>> wrote:
>>>> Hey Jon,
>>>>
>>>> I'm not seeing anything in ProtobuftoPig that is specific to handling
>> protobuf byte fields containing serialized protobuf.
>>>>
>>>> Do you have a Pig example?
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>> On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:
>>>>
>>>>> Look up the util class ProtobuftoPig
>>>>>
>>>>> 2012/3/26 Benjamin Juhn <be...@gmail.com>
>>>>>
>>>>>> Thanks Jon,
>>>>>>
>>>>>> I'm using elephant-bird, and it allows me to load message A. I'm not
>> sure
>>>>>> if it has pig bindings allowing me to deserialize message B from
>> message A
>>>>>> though.
>>>>>>
>>>>>> Thanks,
>>>>>> Ben
>>>>>>
>>>>>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
>>>>>>
>>>>>>> Ben,
>>>>>>>
>>>>>>> take a look at elephantbird
>> https://github.com/kevinweil/elephant-bird
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>> 2012/3/26 Benjamin Juhn <be...@gmail.com>
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I'm using pig with protobuf and I have some byte fields containing
>>>>>>>> serialized protobuf data. Is it possible to handle this nested
>>>>>> serialized
>>>>>>>> data with pig?
>>>>>>>>
>>>>>>>> ex.
>>>>>>>> message A {
>>>>>>>> required bytes data = 1 // serialized message B
>>>>>>>> }
>>>>>>>> message B {
>>>>>>>> ...
>>>>>>>> }
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ben
>>>>>>
>>>>>>
>>>>
>>
>>
Re: deserializing nested protobufs
Posted by Raghu Angadi <an...@gmail.com>.
extension are not supported yet. there is a patch pending :
https://github.com/kevinweil/elephant-bird/pull/143
Can you check if that covers your use case?
On Tue, Apr 3, 2012 at 4:32 PM, Benjamin Juhn <be...@gmail.com> wrote:
> Thanks Dmitriy. Doesn't look like that class supports extensions. Am I
> missing something?
>
> - Ben
>
> On Mar 27, 2012, at 10:01 PM, Dmitriy Ryaboy wrote:
>
> > I think you want ProtobufBytesToTuple
> > (
> https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/pig/piggybank/ProtobufBytesToTuple.java
> )
> >
> > This uses ProtobufToPig to do the actual conversion, and wraps it in a
> UDF.
> >
> > D
> >
> > On Tue, Mar 27, 2012 at 7:21 PM, Benjamin Juhn <be...@gmail.com>
> wrote:
> >> Hey Jon,
> >>
> >> I'm not seeing anything in ProtobuftoPig that is specific to handling
> protobuf byte fields containing serialized protobuf.
> >>
> >> Do you have a Pig example?
> >>
> >> Thanks,
> >> Ben
> >>
> >> On Mar 26, 2012, at 4:14 PM, Jonathan Coveney wrote:
> >>
> >>> Look up the util class ProtobuftoPig
> >>>
> >>> 2012/3/26 Benjamin Juhn <be...@gmail.com>
> >>>
> >>>> Thanks Jon,
> >>>>
> >>>> I'm using elephant-bird, and it allows me to load message A. I'm not
> sure
> >>>> if it has pig bindings allowing me to deserialize message B from
> message A
> >>>> though.
> >>>>
> >>>> Thanks,
> >>>> Ben
> >>>>
> >>>> On Mar 26, 2012, at 4:09 PM, Jonathan Coveney wrote:
> >>>>
> >>>>> Ben,
> >>>>>
> >>>>> take a look at elephantbird
> https://github.com/kevinweil/elephant-bird
> >>>>>
> >>>>> Jon
> >>>>>
> >>>>> 2012/3/26 Benjamin Juhn <be...@gmail.com>
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> I'm using pig with protobuf and I have some byte fields containing
> >>>>>> serialized protobuf data. Is it possible to handle this nested
> >>>> serialized
> >>>>>> data with pig?
> >>>>>>
> >>>>>> ex.
> >>>>>> message A {
> >>>>>> required bytes data = 1 // serialized message B
> >>>>>> }
> >>>>>> message B {
> >>>>>> ...
> >>>>>> }
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Ben
> >>>>
> >>>>
> >>
>
>