You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Lukas Nalezenec <lu...@firma.seznam.cz> on 2015/03/25 17:28:55 UTC
How to read Protobuffers in Hive
Hi,
I am trying to write Serde + ObjectInspectors for reading Protobuffers
in Hive.
I tried to use class ProtocolBuffersStructObjectInspector from Hive but
it last worked with old protobuffer version 2.3.
I tried to use ObjectInspector from Twitter Elephant-bird but it does
not work too.
It looks like that it could help if I havent used DynamicMessage$Builder
. The problem is that DynamicMessage$Builder cannot be reused.
When message is build from builder the builder field "fields" is set tu
null and it throws NPE on second build() call.
...
at com.lukas.AbstractProtobufStructObjectInspector.setStructFieldData_default(AbstractProtobufStructObjectInspector.java:253)
at com.lukas.ProtobufStructObjectInspector.setStructFieldData(ProtobufStructObjectInspector.java:61)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:325)
at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:324)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:630)
... 9 more
Caused by: java.lang.NullPointerException
at com.google.protobuf.DynamicMessage$Builder.clearField(DynamicMessage.java:386)
at com.google.protobuf.DynamicMessage$Builder.clearField(DynamicMessage.java:252)
at com.lukas.AbstractProtobufStructObjectInspector.setStructFieldData_default(AbstractProtobufStructObjectInspector.java:177)
... 13 more
I am using Hive 10 but I am also interested in solution for Hive 13.
Does anybody have working Protobuffer Serde ?
Thanks
Best Regards
Lukas
Re: How to read Protobuffers in Hive
Posted by Edward Capriolo <ed...@gmail.com>.
You may be able to use:
https://github.com/edwardcapriolo/hive-protobuf
(Use the branch not master)
This code is based on the avro support. It works well even with nested
objects.
On Wed, Mar 25, 2015 at 12:28 PM, Lukas Nalezenec <
lukas.nalezenec@firma.seznam.cz> wrote:
> Hi,
> I am trying to write Serde + ObjectInspectors for reading Protobuffers in
> Hive.
> I tried to use class ProtocolBuffersStructObjectInspector from Hive but it
> last worked with old protobuffer version 2.3.
> I tried to use ObjectInspector from Twitter Elephant-bird but it does not
> work too.
>
> It looks like that it could help if I havent used DynamicMessage$Builder .
> The problem is that DynamicMessage$Builder cannot be reused.
> When message is build from builder the builder field "fields" is set tu
> null and it throws NPE on second build() call.
>
> ...
> at com.lukas.AbstractProtobufStructObjectInspector.setStructFieldData_default(AbstractProtobufStructObjectInspector.java:253)
> at com.lukas.ProtobufStructObjectInspector.setStructFieldData(ProtobufStructObjectInspector.java:61)
> at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:325)
> at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters$StructConverter.convert(ObjectInspectorConverters.java:324)
> at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:630)
> ... 9 more
> Caused by: java.lang.NullPointerException
> at com.google.protobuf.DynamicMessage$Builder.clearField(DynamicMessage.java:386)
> at com.google.protobuf.DynamicMessage$Builder.clearField(DynamicMessage.java:252)
> at com.lukas.AbstractProtobufStructObjectInspector.setStructFieldData_default(AbstractProtobufStructObjectInspector.java:177)
> ... 13 more
>
>
>
> I am using Hive 10 but I am also interested in solution for Hive 13.
>
> Does anybody have working Protobuffer Serde ?
>
> Thanks
> Best Regards
>
> Lukas
>