[jira] [Commented] (AVRO-1360) C++ Resolving decoder is not working when reader schema has more fields than writer schema

["Steve Roehrs (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-1360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782541#comment-13782541 ] 

Steve Roehrs commented on AVRO-1360:
------------------------------------

First - thanks for the work in getting this part of Avro working for C++. It always appears to be the poor cousin of the Java implementation, and I appreciate the time that you've put into this patch (especially after trying to tackle it myself once, and failing!)

Basic schema resolution seems to working fine for me now, using default values. More troubling to me is that I think there is an issue with correct resolution of enums.  

My writer schema has a type like this :

{
"type" : "enum",
"name" : "foo",
"symbols" : [
  "charlie",
  "delta",
  "echo"
]
}

The reader schema looks like this ;
{
"type" : "enum",
"name" : "foo",
"symbols" : [
  "alpha",
  "bravo",
  "charlie",
  "delta",
  "echo"
]
}

When I write a value 'charlie' to my data file, and read it back using the resolving decoder (DataFileReader<avro::GenericDatum> reader(filename, readerSchema) - I get the value 'alpha' not the 'charlie'  that I expect. 

GenericEnum::value() returns a 0, and GenericEnum::symbol() returns 'alpha' - so it's clearly using the reader schema, but not remapping the value as part of the schema resolution process. 

Sorry I don't have a full test case I can send you, I'm unable to export anything from my development system.

Should this be raised as a separate issue?


> C++ Resolving decoder is not working when reader schema has more fields than writer schema
> ------------------------------------------------------------------------------------------
>
>                 Key: AVRO-1360
>                 URL: https://issues.apache.org/jira/browse/AVRO-1360
>             Project: Avro
>          Issue Type: Bug
>          Components: c++
>    Affects Versions: 1.7.4
>            Reporter: Ramana Suvarapu
>            Assignee: Thiruvalluvan M. G.
>         Attachments: AVRO-1360-2.patch, AVRO-1360-3.patch, AVRO-1360.patch, testreader, testreader-1, testreader.hh, testwriter, testwriter-1, testwriter.hh
>
>
> When reader schema has more number of fields than writer schema, C++ implementation of resolving decoder is throwing exception "throwing exception "Don't know how to handle excess fields for reader.” with out checking whether fields are optional or fields have default values.
> Attached are reader and writer schemas. Record in reader schema has 2 additional fields than writer schema. One field is required field but it has default value and another one is optional field (union of null and string). Since one has default value and another is optional both reader and writer schemas are supposed to be compatible. 
>  
> {"name": "defaultField", "type": "string", "default": "DEFAULT", "declared":"true"},     
> {"name": "optionalField", "type": ["string", "null"],"declared":"true"},
>  
> main()
> {
>   avro::ValidSchema readerSchema = load("reader.json");
>   avro::ValidSchema writerSchema = load("writer.json");
>   avro::DecoderPtr d = avro::resolvingDecoder(writerSchema, readerSchema,avro::binaryDecoder());
> }
>  
> But when I tried to create resolving decoder, I am getting "Don't know how to handle excess fields for reader.” But Java implementation works.  
>  
> Can you please let us know if there are any other limitations with c++ implementation of ResolvingDecoder? We are planning to use it in our project and we want to make sure it works as per avro specification.



--
This message was sent by Atlassian JIRA
(v6.1#6144)