You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Rui Martins <ru...@gmail.com> on 2013/05/13 14:14:39 UTC
Trying to write a custom HiveOutputFormat
Hi guys,
I'm currently writing my on HiveOutputFormat as I would like to write the
output of hive queries into a specific protobuf format my team is using.
I have managed to do this however, the Writable object I get from Hive as a
result of a SELECT query is of type Text. This means that I have to split
the string to find my fields but that's very error prone, specially if some
fields are strings that may contain spaces.
My question is:
1) How do I get a Hive Writable that gives me each field of each result row?
Thank you,
rui
Re: Trying to write a custom HiveOutputFormat
Posted by Ramki Gmail <ra...@gmail.com>.
You may need to implement a custom SerDe.
Regards,
Ramki
On May 13, 2013, at 5:14 AM, Rui Martins <ru...@gmail.com> wrote:
> Hi guys,
>
> I'm currently writing my on HiveOutputFormat as I would like to write the output of hive queries into a specific protobuf format my team is using.
> I have managed to do this however, the Writable object I get from Hive as a result of a SELECT query is of type Text. This means that I have to split the string to find my fields but that's very error prone, specially if some fields are strings that may contain spaces.
>
> My question is:
> 1) How do I get a Hive Writable that gives me each field of each result row?
>
> Thank you,
> rui
Re: Trying to write a custom HiveOutputFormat
Posted by Rui Martins <ru...@gmail.com>.
Cool, thanks for the clarification guys. I'll get on with the
implementation of another SerDe. :)
@Edward - Thanks for the link. I actually already have a protobuffer SerDe
that I implemented that allows passing any protobuffer schema into it (as
long as it's proto 2.4.1) and also has some other features like mimic'ing
Maps that proto doesn't have.
I'll reply back to this thread with my progress.
Thanks guys,
rui
On Mon, May 13, 2013 at 2:54 PM, Edward Capriolo <ed...@gmail.com>wrote:
> You need to use a combination of output format and serde, this might allow
> you to do something like present struct objects to the input format rather
> then Text objects.
>
> You may want to take a look at the protobuf input format we use:
> https://github.com/edwardcapriolo/hive-protobuf/
>
> You could reverse the logic here and design an output format.
>
>
> On Mon, May 13, 2013 at 8:14 AM, Rui Martins <ru...@gmail.com>wrote:
>
>> Hi guys,
>>
>> I'm currently writing my on HiveOutputFormat as I would like to write the
>> output of hive queries into a specific protobuf format my team is using.
>> I have managed to do this however, the Writable object I get from Hive as
>> a result of a SELECT query is of type Text. This means that I have to split
>> the string to find my fields but that's very error prone, specially if some
>> fields are strings that may contain spaces.
>>
>> My question is:
>> 1) How do I get a Hive Writable that gives me each field of each result
>> row?
>>
>> Thank you,
>> rui
>>
>
>
Re: Trying to write a custom HiveOutputFormat
Posted by Owen O'Malley <om...@apache.org>.
You could also look at the OrcSerde and how it works.
https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcSerde.java
Basically, OrcSerde on "serialize" just wraps the row and object inspector
in a fake writable. That is passed down to the OutputFormat. On
"deserialize" it does the reverse and just passes back the object from the
InputFormat.
-- Owen
On Mon, May 13, 2013 at 6:54 AM, Edward Capriolo <ed...@gmail.com>wrote:
> You need to use a combination of output format and serde, this might allow
> you to do something like present struct objects to the input format rather
> then Text objects.
>
> You may want to take a look at the protobuf input format we use:
> https://github.com/edwardcapriolo/hive-protobuf/
>
> You could reverse the logic here and design an output format.
>
>
> On Mon, May 13, 2013 at 8:14 AM, Rui Martins <ru...@gmail.com>wrote:
>
>> Hi guys,
>>
>> I'm currently writing my on HiveOutputFormat as I would like to write the
>> output of hive queries into a specific protobuf format my team is using.
>> I have managed to do this however, the Writable object I get from Hive as
>> a result of a SELECT query is of type Text. This means that I have to split
>> the string to find my fields but that's very error prone, specially if some
>> fields are strings that may contain spaces.
>>
>> My question is:
>> 1) How do I get a Hive Writable that gives me each field of each result
>> row?
>>
>> Thank you,
>> rui
>>
>
>
Re: Trying to write a custom HiveOutputFormat
Posted by Edward Capriolo <ed...@gmail.com>.
You need to use a combination of output format and serde, this might allow
you to do something like present struct objects to the input format rather
then Text objects.
You may want to take a look at the protobuf input format we use:
https://github.com/edwardcapriolo/hive-protobuf/
You could reverse the logic here and design an output format.
On Mon, May 13, 2013 at 8:14 AM, Rui Martins <ru...@gmail.com> wrote:
> Hi guys,
>
> I'm currently writing my on HiveOutputFormat as I would like to write the
> output of hive queries into a specific protobuf format my team is using.
> I have managed to do this however, the Writable object I get from Hive as
> a result of a SELECT query is of type Text. This means that I have to split
> the string to find my fields but that's very error prone, specially if some
> fields are strings that may contain spaces.
>
> My question is:
> 1) How do I get a Hive Writable that gives me each field of each result
> row?
>
> Thank you,
> rui
>