You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@crunch.apache.org by Kaspar Fischer <ka...@dreizak.com> on 2014/11/05 07:50:12 UTC
POJO serialization using generics
Hi everybody,
I am using Crunch with Avros serialization. Some intermediate objects of my
pipeline are POJO's that have fields with generics. When running the
pipeline I get:
Exception in thread "main" org.apache.avro.AvroTypeException: Unknown type:
T
at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:220)
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374)
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283)
at
org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
at
org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354)
at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:197)
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374)
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283)
at
org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
at
org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354)
at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:154)
at org.apache.crunch.types.avro.Avros.reflects(Avros.java:275)
at org.apache.crunch.types.avro.Avros.containers(Avros.java:264)
at org.apache.crunch.types.avro.Avros.records(Avros.java:252)
How can I get around this? Is there a way to serialize the object using
Java serialization (or Kryo)?
Thanks for any pointers,
Kaspar
Re: POJO serialization using generics
Posted by Kaspar Fischer <ka...@dreizak.com>.
Hey Josh,
Great, very useful!
Thanks a lot for sharing this.
Cheers,
Kaspar
> On 04.11.2014, at 23:13, Josh Wills <jw...@cloudera.com> wrote:
>
> Hey Kaspar,
>
> I wrote a PType for supporting arbitrarily serializables as part of some work I was doing on Oryx:
>
> https://github.com/cloudera/oryx/blob/master/computation-common/src/main/java/com/cloudera/oryx/computation/common/types/Serializables.java
>
> I never committed it to Crunch proper b/c it's not the sort of thing we try to encourage people to do, though like you, I have been in situations where it was the most expedient solution. ;-)
>
> J
>
>> On Tue, Nov 4, 2014 at 10:50 PM, Kaspar Fischer <ka...@dreizak.com> wrote:
>> Hi everybody,
>>
>> I am using Crunch with Avros serialization. Some intermediate objects of my pipeline are POJO's that have fields with generics. When running the pipeline I get:
>>
>> Exception in thread "main" org.apache.avro.AvroTypeException: Unknown type: T
>> at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:220)
>> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374)
>> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283)
>> at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
>> at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
>> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354)
>> at org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:197)
>> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374)
>> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283)
>> at org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
>> at org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
>> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354)
>> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:154)
>> at org.apache.crunch.types.avro.Avros.reflects(Avros.java:275)
>> at org.apache.crunch.types.avro.Avros.containers(Avros.java:264)
>> at org.apache.crunch.types.avro.Avros.records(Avros.java:252)
>>
>> How can I get around this? Is there a way to serialize the object using Java serialization (or Kryo)?
>>
>> Thanks for any pointers,
>> Kaspar
>
>
>
> --
> Director of Data Science
> Cloudera
> Twitter: @josh_wills
Re: POJO serialization using generics
Posted by Josh Wills <jw...@cloudera.com>.
Hey Kaspar,
I wrote a PType for supporting arbitrarily serializables as part of some
work I was doing on Oryx:
https://github.com/cloudera/oryx/blob/master/computation-common/src/main/java/com/cloudera/oryx/computation/common/types/Serializables.java
I never committed it to Crunch proper b/c it's not the sort of thing we try
to encourage people to do, though like you, I have been in situations where
it was the most expedient solution. ;-)
J
On Tue, Nov 4, 2014 at 10:50 PM, Kaspar Fischer <ka...@dreizak.com>
wrote:
> Hi everybody,
>
> I am using Crunch with Avros serialization. Some intermediate objects of
> my pipeline are POJO's that have fields with generics. When running the
> pipeline I get:
>
> Exception in thread "main" org.apache.avro.AvroTypeException: Unknown
> type: T
>
> at
> org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:220)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283)
>
> at
> org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
>
> at
> org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354)
>
> at
> org.apache.avro.specific.SpecificData.createSchema(SpecificData.java:197)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:374)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:283)
>
> at
> org.apache.avro.reflect.ReflectData.createFieldSchema(ReflectData.java:430)
>
> at
> org.apache.avro.reflect.ReflectData$AllowNull.createFieldSchema(ReflectData.java:72)
>
> at org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:354)
>
> at org.apache.avro.specific.SpecificData.getSchema(SpecificData.java:154)
>
> at org.apache.crunch.types.avro.Avros.reflects(Avros.java:275)
>
> at org.apache.crunch.types.avro.Avros.containers(Avros.java:264)
>
> at org.apache.crunch.types.avro.Avros.records(Avros.java:252)
>
> How can I get around this? Is there a way to serialize the object using
> Java serialization (or Kryo)?
>
> Thanks for any pointers,
> Kaspar
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>