You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Lukas Nalezenec <lu...@firma.seznam.cz> on 2015/03/25 17:28:55 UTC

How to read Protobuffers in Hive

Hi,
I am trying to write Serde + ObjectInspectors for reading Protobuffers 
in Hive.
I tried to use class ProtocolBuffersStructObjectInspector from Hive but 
it last worked with old protobuffer version 2.3.
I tried to use ObjectInspector from Twitter Elephant-bird but it does 
not work too.

It looks like that it could help if I havent used DynamicMessage$Builder 
. The problem is that DynamicMessage$Builder cannot be reused.
When message is build from builder the builder field "fields" is set tu 
null and it throws NPE on second build() call.

...
	at com.lukas.AbstractProtobufStructObjectInspector.setStructFieldData_default(AbstractProtobufStructObjectInspector.java:253)
	at com.lukas.ProtobufStructObjectInspector.setStructFieldData(ProtobufStructObjectInspector.java:61)
	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:325)
	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:324)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:630)
	... 9 more
Caused by: java.lang.NullPointerException
	at com.google.protobuf.DynamicMessage$Builder.clearField(DynamicMessage.java:386)
	at com.google.protobuf.DynamicMessage$Builder.clearField(DynamicMessage.java:252)
	at com.lukas.AbstractProtobufStructObjectInspector.setStructFieldData_default(AbstractProtobufStructObjectInspector.java:177)
	... 13 more



I am using Hive 10 but I am also interested in solution for Hive 13.

Does anybody have working Protobuffer Serde ?

Thanks
Best Regards

Lukas

Re: How to read Protobuffers in Hive

Posted by Edward Capriolo <ed...@gmail.com>.
You may be able to use:
https://github.com/edwardcapriolo/hive-protobuf

(Use the branch not master)

This code is based on the avro support. It works well even with nested
objects.




On Wed, Mar 25, 2015 at 12:28 PM, Lukas Nalezenec <
lukas.nalezenec@firma.seznam.cz> wrote:

>  Hi,
> I am trying to write Serde + ObjectInspectors for reading Protobuffers in
> Hive.
> I tried to use class ProtocolBuffersStructObjectInspector from Hive but it
> last worked with old protobuffer version 2.3.
> I tried to use ObjectInspector from Twitter Elephant-bird but it does not
> work too.
>
> It looks like that it could help if I havent used DynamicMessage$Builder .
> The problem is that DynamicMessage$Builder cannot be reused.
> When message is build from builder the builder field "fields" is set tu
> null and it throws NPE on second build() call.
>
> ...
> 	at com.lukas.AbstractProtobufStructObjectInspector.setStructFieldData_default(AbstractProtobufStructObjectInspector.java:253)
> 	at com.lukas.ProtobufStructObjectInspector.setStructFieldData(ProtobufStructObjectInspector.java:61)
> 	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:325)
> 	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:324)
> 	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:630)
> 	... 9 more
> Caused by: java.lang.NullPointerException
> 	at com.google.protobuf.DynamicMessage$Builder.clearField(DynamicMessage.java:386)
> 	at com.google.protobuf.DynamicMessage$Builder.clearField(DynamicMessage.java:252)
> 	at com.lukas.AbstractProtobufStructObjectInspector.setStructFieldData_default(AbstractProtobufStructObjectInspector.java:177)
> 	... 13 more
>
>
>
> I am using Hive 10 but I am also interested in solution for Hive 13.
>
> Does anybody have working Protobuffer Serde ?
>
> Thanks
> Best Regards
>
> Lukas
>