Best way to deal with incompatible column type changes

[Udit Mehta <um...@groupon.com>]

Hi,

I have a Hive external table with a lot of partitions where the underlying
data is in JSON. I use this popular serde
<https://github.com/rcongiu/Hive-JSON-Serde> to read and write in JSON
format.

So I have a data stream where sometimes there are changes to the JSON
structure. For eg, a key might change its type from string to a struct or
an array. Replacing/Changing the column via the ALTER statement does not
really help since it results in a ClassCastException(Based on this ticket
<https://issues.apache.org/jira/browse/HIVE-4409>).

My question is what would be the best way to deal with such schema changes
without dropping/creating the table again. Basically i dont want to lose my
partitions.

I am currently using Hive version 0.13.1 but planning to move to version
1.2.1 soon.

Any help/advise would be appreciated.

Thanks,
Udit

Re: Best way to deal with incompatible column type changes

[Manikanda Prabhu <gm...@gmail.com>]

Udit 

Kindly check AVRO file format and its schema evolution 

Mani 

Sent from my iPhone

> On Oct 15, 2015, at 4:50 PM, Udit Mehta <um...@groupon.com> wrote:
> 
> Hi,
> 
> I have a Hive external table with a lot of partitions where the underlying data is in JSON. I use this popular serde to read and write in JSON format.
> 
> So I have a data stream where sometimes there are changes to the JSON structure. For eg, a key might change its type from string to a struct or an array. Replacing/Changing the column via the ALTER statement does not really help since it results in a ClassCastException(Based on this ticket). 
> 
> My question is what would be the best way to deal with such schema changes without dropping/creating the table again. Basically i dont want to lose my partitions.
> 
> I am currently using Hive version 0.13.1 but planning to move to version 1.2.1 soon.
> 
> Any help/advise would be appreciated.
> 
> Thanks,
> Udit