[jira] [Commented] (AVRO-2160) Json to Avro with non required value and union schema failing

["Christophe Le Saec (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-2160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17554992#comment-17554992 ] 

Christophe Le Saec commented on AVRO-2160:
------------------------------------------

Issue is not about "union type" but more about default value, it also occurs with schema :
{code:json}
{"type":"record", "namespace":"foo","name":"Person",
  "fields":[
    {"name":"lastname","type": "string", "default": "last"},
    {"name":"firstname","type":"string"},
    {"name":"age","type":["null","int"], "default":null}
]}
{code}
And input data (so lastname is missing, and default value "last" should replace it) ?
{code:json}
{"firstname":"John","age":{"int":35}}
{code}
I think [FieldAdjustAction Symbol class|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/parsing/Symbol.java#L587] should embed default value that could be easily provided by [JsonGrammarGenerator|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/parsing/JsonGrammarGenerator.java#L85] and then [JsonDecoder could use it (if it has one) before throwing exception|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/io/JsonDecoder.java#L473].
(At least, to my point, that's the meaning of default value)

> Json to Avro with non required value and union schema failing
> -------------------------------------------------------------
>
>                 Key: AVRO-2160
>                 URL: https://issues.apache.org/jira/browse/AVRO-2160
>             Project: Apache Avro
>          Issue Type: Bug
>          Components: java
>    Affects Versions: 1.8.2
>            Reporter: Lydie
>            Priority: Critical
>              Labels: java
>
> I am trying to convert this string:
> str str4
> using this schema:
> {"type":"record", "namespace":"foo","name":"Person","fields":[\\{"name":"lastname","type": ["null","string"], "default":null}
> ,\{"name":"firstname","type":"string"},{"name":"age","type":["null","int"], "default":null}]}
> I get this error 
> {color:#ff0000}com.syapse.messagePublisher.publisher.AvroEncodeException: Expected field name not found: lastnamein\{"firstname":"John","age":{"int":35}}{color}at com.syapse.messagePublisher.publisher.AvroEncoder.convertJsonToAvro(AvroEncoder.java:78)
>  
> Although this should give me the correct syntax for a non required filed.
> Note that it works for 
> {"lastname":\\{"string" : "Doe"}
> ,"firstname":"John","age":\{"int":36}}
>  
> What am I missing ( using Abro 1.8.2)
> here is my code:
>  
> {code:java}
> public static byte[] convertJsonToAvro(byte[] data, String schemaStr) throws AvroEncodeException {
> InputStream input = null;
> DataFileWriter<GenericRecord> writer = null;
> ByteArrayOutputStream output = null;
> try {
> Schema schema = new Schema.Parser().parse(schemaStr);
> DatumReader<GenericRecord> reader = new GenericDatumReader<GenericRecord>(schema);
> input = new ByteArrayInputStream(data);
> DataInputStream din = new DataInputStream(input);
> output = new ByteArrayOutputStream();
> writer = new DataFileWriter<GenericRecord>(new GenericDatumWriter<GenericRecord>());
> writer.create(schema, output);
> Decoder decoder = DecoderFactory.get().jsonDecoder(schema, din);
> GenericRecord datum = null;
> while (true) {
> try {
> datum = reader.read(null, decoder);
> } catch (EOFException eofe) {
> break;
> }
> writer.append(datum);
> }
> writer.flush();
> writer.close();
> return output.toByteArray();
> } catch (AvroTypeException e) {
> throw new AvroEncodeException(e.getMessage() + "in" + new String(data));
> } catch (IOException e1) {
> throw new AvroEncodeException("Error decoding Json " + e1.getMessage());
> } finally {
> try {
> input.close();
> } catch (Exception e) {
> }
> }
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)