You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@avro.apache.org by Doug Cutting <cu...@apache.org> on 2009/08/01 01:56:07 UTC

Re: Unknown type java.util.List with ReflectData.getSchema

Avro uses org.apache.avro.util.Utf8 to represent strings, not 
java.lang.String.

Doug

tazan007 wrote:
> Hi guys, I am getting this error when trying to build a schema using an 
> existing class with ReflectData.getSchema method with a List<String> 
> field.  Is this expected behaviour?  Are List's not supported?
> 
> ptype = java.util.List<java.lang.String> raw = interface java.util.List
> param =class java.lang.String
> Exception in thread "main" org.apache.avro.AvroTypeException: Unknown 
> type: java.util.List<java.lang.String>
>     at 
> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:192)
>     at 
> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:185)
>     at org.apache.avro.reflect.ReflectData.getSchema(ReflectData.java:107)
> 
> Thanks,
> Hiral

Re: Unknown type java.util.List with ReflectData.getSchema

Posted by Doug Cutting <cu...@apache.org>.
Scott Carey wrote:
> I don’t know what the best way to support that is, but it is important 
> IMO to have a way to control how the default types map to Java classes 
> via reflection and have a couple built-in options for the most common 
> variations.  Subclassing works, a pluggable type map probably would as well.

Would you like to file a Jira issue for this?  Thanks!

Doug

Re: Unknown type java.util.List with ReflectData.getSchema

Posted by Scott Carey <sc...@richrelevance.com>.
The use case I will definitely need is one where a legacy class and a new class share the same Avro schema.  These two classes have  to coexist for a while, sometimes in the same process.  The legacy class maps various Java types to avro types differently (String, List, Date are some more obvious ones) and cannot be changed.   In the long run the new class will be used exclusively, which can do things like lazily switch from Utf8 to transient String once an operation is requested that needs String and on serialization knows how to deal with updating persistent types appropriately.

I don't know what the best way to support that is, but it is important IMO to have a way to control how the default types map to Java classes via reflection and have a couple built-in options for the most common variations.  Subclassing works, a pluggable type map probably would as well.


On 8/6/09 9:28 PM, "Doug Cutting" <cu...@apache.org> wrote:

tazan007 wrote:
> Yea makes sense.  I am trying to go from Java to Avro :D so I ended up
> overriding getArraySize, getArrayElements, writeRecord, and writeString
> from ReflectDatumWriter so I could have it convert String to Utf8, Date
> to Long, and List to GenericArray.

Please consider contributing this.  File an issue in Jira and attach the
patch.

http://wiki.apache.org/hadoop/Avro/HowToContribute

> Also had to make some changes to ReflectData's createSchema to support
> String and Lists  and changed RecordSchema's fieldsToJson to translate
> Date objects to long.  Basically a cluge but works for getting data out
> of my Java objects into Avro objects.

Perhaps we should make ReflectData extensible, so that subclasses can
determine the implementation types?

Doug

Re: Unknown type java.util.List with ReflectData.getSchema

Posted by Doug Cutting <cu...@apache.org>.
tazan007 wrote:
> Yea makes sense.  I am trying to go from Java to Avro :D so I ended up 
> overriding getArraySize, getArrayElements, writeRecord, and writeString 
> from ReflectDatumWriter so I could have it convert String to Utf8, Date 
> to Long, and List to GenericArray. 

Please consider contributing this.  File an issue in Jira and attach the 
patch.

http://wiki.apache.org/hadoop/Avro/HowToContribute

> Also had to make some changes to ReflectData's createSchema to support 
> String and Lists  and changed RecordSchema's fieldsToJson to translate 
> Date objects to long.  Basically a cluge but works for getting data out 
> of my Java objects into Avro objects.

Perhaps we should make ReflectData extensible, so that subclasses can 
determine the implementation types?

Doug

Re: Unknown type java.util.List with ReflectData.getSchema

Posted by tazan007 <ta...@gmail.com>.
Yea makes sense.  I am trying to go from Java to Avro :D so I ended up
overriding getArraySize, getArrayElements, writeRecord, and writeString from
ReflectDatumWriter so I could have it convert String to Utf8, Date to Long,
and List to GenericArray.

Also had to make some changes to ReflectData's createSchema to support
String and Lists  and changed RecordSchema's fieldsToJson to translate Date
objects to long.  Basically a cluge but works for getting data out of my
Java objects into Avro objects.

If anyone is interested in doing the same, let me know.

-Hiral

On Thu, Aug 6, 2009 at 2:27 PM, Doug Cutting <cu...@apache.org> wrote:

> Sorry, List<String> is not supported, rather one must use
> GenericArray<Utf8>.  The reason is that both the specific and the reflect
> implementations inherit from the generic implementation, primarily just
> overriding the implementation of records.  So the mapping from Avro to Java
> (excepting records) is described at:
>
>
> http://hadoop.apache.org/avro/docs/current/api/java/org/apache/avro/generic/package-summary.html
>
> The GenericArray interface is mostly a subset of the List interface, but
> includes one non-List method, peek(), in order to permit reuse of instances.
>
> Doug
>
> tazan007 wrote:
>
>> Still getting similar error with Utf8:
>>
>> ptype = java.util.List<org.apache.avro.util.Utf8> raw = interface
>> java.util.List
>> param =class org.apache.avro.util.Utf8
>> Exception in thread "main" org.apache.avro.AvroTypeException: Unknown
>> type: java.util.List<org.apache.avro.util.Utf8>
>>    at
>> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:192)
>>    at
>> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:185)
>>    at org.apache.avro.reflect.ReflectData.getMessage(ReflectData.java:230)
>>    at
>> org.apache.avro.reflect.ReflectData.getProtocol(ReflectData.java:206)
>>
>> Are lists not support?
>>
>> -Hiral
>>
>> On Fri, Jul 31, 2009 at 4:56 PM, Doug Cutting <cutting@apache.org<mailto:
>> cutting@apache.org>> wrote:
>>
>>    Avro uses org.apache.avro.util.Utf8 to represent strings, not
>>    java.lang.String.
>>
>>    Doug
>>
>>
>>    tazan007 wrote:
>>
>>        Hi guys, I am getting this error when trying to build a schema
>>        using an existing class with ReflectData.getSchema method with a
>>        List<String> field.  Is this expected behaviour?  Are List's not
>>        supported?
>>
>>        ptype = java.util.List<java.lang.String> raw = interface
>>        java.util.List
>>        param =class java.lang.String
>>        Exception in thread "main" org.apache.avro.AvroTypeException:
>>        Unknown type: java.util.List<java.lang.String>
>>           at
>>
>>  org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:192)
>>           at
>>
>>  org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:185)
>>           at
>>        org.apache.avro.reflect.ReflectData.getSchema(ReflectData.java:107)
>>
>>        Thanks,
>>        Hiral
>>
>>
>>

Re: Unknown type java.util.List with ReflectData.getSchema

Posted by Doug Cutting <cu...@apache.org>.
Sorry, List<String> is not supported, rather one must use 
GenericArray<Utf8>.  The reason is that both the specific and the 
reflect implementations inherit from the generic implementation, 
primarily just overriding the implementation of records.  So the mapping 
from Avro to Java (excepting records) is described at:

http://hadoop.apache.org/avro/docs/current/api/java/org/apache/avro/generic/package-summary.html

The GenericArray interface is mostly a subset of the List interface, but 
includes one non-List method, peek(), in order to permit reuse of instances.

Doug

tazan007 wrote:
> Still getting similar error with Utf8:
> 
> ptype = java.util.List<org.apache.avro.util.Utf8> raw = interface 
> java.util.List
> param =class org.apache.avro.util.Utf8
> Exception in thread "main" org.apache.avro.AvroTypeException: Unknown 
> type: java.util.List<org.apache.avro.util.Utf8>
>     at 
> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:192)
>     at 
> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:185)
>     at org.apache.avro.reflect.ReflectData.getMessage(ReflectData.java:230)
>     at org.apache.avro.reflect.ReflectData.getProtocol(ReflectData.java:206)
> 
> Are lists not support?
> 
> -Hiral
> 
> On Fri, Jul 31, 2009 at 4:56 PM, Doug Cutting <cutting@apache.org 
> <ma...@apache.org>> wrote:
> 
>     Avro uses org.apache.avro.util.Utf8 to represent strings, not
>     java.lang.String.
> 
>     Doug
> 
> 
>     tazan007 wrote:
> 
>         Hi guys, I am getting this error when trying to build a schema
>         using an existing class with ReflectData.getSchema method with a
>         List<String> field.  Is this expected behaviour?  Are List's not
>         supported?
> 
>         ptype = java.util.List<java.lang.String> raw = interface
>         java.util.List
>         param =class java.lang.String
>         Exception in thread "main" org.apache.avro.AvroTypeException:
>         Unknown type: java.util.List<java.lang.String>
>            at
>         org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:192)
>            at
>         org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:185)
>            at
>         org.apache.avro.reflect.ReflectData.getSchema(ReflectData.java:107)
> 
>         Thanks,
>         Hiral
> 
>

Re: Unknown type java.util.List with ReflectData.getSchema

Posted by tazan007 <ta...@gmail.com>.
Still getting similar error with Utf8:

ptype = java.util.List<org.apache.avro.util.Utf8> raw = interface
java.util.List
param =class org.apache.avro.util.Utf8
Exception in thread "main" org.apache.avro.AvroTypeException: Unknown type:
java.util.List<org.apache.avro.util.Utf8>
    at
org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:192)
    at
org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:185)
    at org.apache.avro.reflect.ReflectData.getMessage(ReflectData.java:230)
    at org.apache.avro.reflect.ReflectData.getProtocol(ReflectData.java:206)

Are lists not support?

-Hiral

On Fri, Jul 31, 2009 at 4:56 PM, Doug Cutting <cu...@apache.org> wrote:

> Avro uses org.apache.avro.util.Utf8 to represent strings, not
> java.lang.String.
>
> Doug
>
>
> tazan007 wrote:
>
>> Hi guys, I am getting this error when trying to build a schema using an
>> existing class with ReflectData.getSchema method with a List<String> field.
>>  Is this expected behaviour?  Are List's not supported?
>>
>> ptype = java.util.List<java.lang.String> raw = interface java.util.List
>> param =class java.lang.String
>> Exception in thread "main" org.apache.avro.AvroTypeException: Unknown
>> type: java.util.List<java.lang.String>
>>    at
>> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:192)
>>    at
>> org.apache.avro.reflect.ReflectData.createSchema(ReflectData.java:185)
>>    at org.apache.avro.reflect.ReflectData.getSchema(ReflectData.java:107)
>>
>> Thanks,
>> Hiral
>>
>