[jira] [Updated] (AVRO-1500) Unknown datum type exception during union type resolution (no short to int conversion).

["Michael Pershyn (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AVRO-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Pershyn updated AVRO-1500:
----------------------------------

    Labels: easyfix newbie patch  (was: )
    Status: Patch Available  (was: Open)

Added a patch that have fixed this problem in our case. With this patch the conversion of thrift-objects with optional int16 fields is correct.
Also, isNumber check was not added because Long is a different type, so only two cases should be checked.
Please review :-)

> Unknown datum type exception during union type resolution (no short to int conversion).
> ---------------------------------------------------------------------------------------
>
>                 Key: AVRO-1500
>                 URL: https://issues.apache.org/jira/browse/AVRO-1500
>             Project: Avro
>          Issue Type: Bug
>    Affects Versions: 1.7.6, 1.7.5, 1.7.4
>         Environment: java API
>            Reporter: Michael Pershyn
>              Labels: easyfix, patch, newbie
>
> There is a conversion for values of type {{short}} (and other numeric types) if in the schema they are declared as int:
> {code:title=/lang/java/avro/src/main/java/org/apache/avro/generic/GenericDatumWriter.java}
>  protected void write(Schema schema, Object datum, Encoder out)
>     throws IOException {
>     try {
>       switch (schema.getType()) {
>         // ...
>         case INT:     out.writeInt(((Number)datum).intValue()); break;
>         // ...
> {code}
> So, if a value of {{short}} type is passed to INT field, it will be converted and saved as INT in avro.
> But, when there is next field in schema: 
> {code}
>  ["null",{"type":"int","thrift":"short"}] {code}
> which is a union with int in it, and short is passed in (lets say 5), then we are having the next exception:
> {code}
> org.apache.avro.AvroRuntimeException: Unknown datum type: 5
> 	at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:593)
> 	at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:558)
> 	at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:144) 
> {code}
> This happens because in {{org.apache.avro.generic.GenericData}} there is no check if the passed object has a type of {{java.lang.Short}}, and it is not get converted then in {{write}} method of {{GenericDatumWriter}}:
> {code:title=lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java}
> /** Return the schema full name for a datum.  Called by {@link
>    * #resolveUnion(Schema,Object)}. */
> protected String getSchemaName(Object datum) {
> /* ... */
> if (isInteger(datum))
>   return Type.INT.getName();
> if (isLong(datum))
>   return Type.LONG.getName();
>  if (isFloat(datum))
>  return Type.FLOAT.getName();
> if (isDouble(datum))
>   return Type.DOUBLE.getName();
>  if (isBoolean(datum))
>   return Type.BOOLEAN.getName();
> throw new AvroRuntimeException("Unknown datum type: "+datum);
> {code}
> This error initially occured during thrift to avro conversion, when thrift obj had optional field of type i16.
> In thrift to avro schema converter, if the type is short in thrift (i16) it will be implicitly converted to int in avro-schema, so the values should be converted as well. This is already done if they are not in the union (not optional). Otherwise the exception is thrown.
> The snippet from schema conversion code is below:
> {code:title=lang/java/thrift/src/main/java/org/apache/avro/thrift/ThriftData.java}
> private Schema getSchema(FieldValueMetaData f) {
>   switch (f.type) {
>   /* ... */ 
>     case TType.I16:
>       Schema s = Schema.create(Schema.Type.INT);
>       s.addProp(THRIFT_PROP, "short");
>     return s;
>    /* ... */
> {code}
> Proposal is to add isShort check to generic data, as well as isShort method implementation:
> {code:title=lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java}
> protected String getSchemaName(Object datum) {
> // ..
> if (isInteger(datum) || isShort(datum))
>    return Type.INT.getName();
> // ..
> {code}
> or maybe even some kind of isNumeric method, so the behaviour will be same for INT fields and INT fields that are in Union.



--
This message was sent by Atlassian JIRA
(v6.2#6252)