How to Use AvroStorage to read avro data and convert into a new (compatible) schema

[Runping Qi <ru...@gmail.com>]

Hi,

I have the following problem to solve:

I have data in HDFS written out using AvroStorage with schema1.
I have schema2 which is different than schema1 but they are compatible.
Now, I want to use AvroStorage to load the data and convert to schema2.
How can I achieve that?

I tried something like:

    data = load 'my_data_dir' using AvroStorage('schema', SCHEMA2_STRING);
    describe data;

The schema for the loaded-in data is still in schema1, not in schema2.

Any help is highly appreciated.

Thanks,

Runping Qi
A9.COM

Re: How to Use AvroStorage to read avro data and convert into a new (compatible) schema

[Russell Jurney <ru...@gmail.com>]

Try loading the data in schema one, and storing it/transforming it in
schema two in a new location?

Then try to load it, and see what it is. This may not be feasible if
your data is large, but worth a try to help debug?

Russell Jurney http://datasyndrome.com

On Oct 5, 2012, at 7:48 PM, Runping Qi <ru...@gmail.com> wrote:

> Hi,
>
> I have the following problem to solve:
>
> I have data in HDFS written out using AvroStorage with schema1.
> I have schema2 which is different than schema1 but they are compatible.
> Now, I want to use AvroStorage to load the data and convert to schema2.
> How can I achieve that?
>
> I tried something like:
>
>    data = load 'my_data_dir' using AvroStorage('schema', SCHEMA2_STRING);
>    describe data;
>
> The schema for the loaded-in data is still in schema1, not in schema2.
>
> Any help is highly appreciated.
>
> Thanks,
>
> Runping Qi
> A9.COM