[GitHub] [pulsar] shiv4289 edited a comment on issue #6419: auto_consume for avro crashes in decode (updated producer .class, old consumer schema .class)

[GitBox <gi...@apache.org>]

shiv4289 edited a comment on issue #6419: auto_consume for avro crashes in decode (updated producer .class, old consumer schema .class)
URL: https://github.com/apache/pulsar/issues/6419#issuecomment-591270268
 
 
   On debugging more, the issue seems to be this:
   1. We maintain a library (say, _avro-schemas_) that contains all .avsc files and their generated .class files. Lets say we are on version v1 of this library on producer as well as consumer applications. This works fine.
   2. We add a new field (with defaults) to avro in _avro-schemas_, update our library to v2 and update this in the producer application.
   3. Consumer application is still using v1 of the _avro-schemas_ library.
   4. The apache avro library looks for a class name with the name of schema in the class loader context. In our case, it finds the old class in v1.
   5. Now [it goes through every field in schema](https://github.com/apache/avro/blob/8199b9626e4badd34a85946cd223a91863f44cee/lang/java/avro/src/main/java/org/apache/avro/generic/GenericDatumReader.java#L235) (updated schema), fetches that from binary data buffer and tries to set that in my old class. The fields in the old class are found by index and then updated by writing to that position ([here](https://github.com/apache/avro/blob/8199b9626e4badd34a85946cd223a91863f44cee/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java#L815)) which does not exist for new fields in the old generated class, thence the crash. 
   
   **Stacktrace**
   
   ```
   [2020-02-26T10:23:34.441Z] ERROR main BeamRuleEngine:51 [/] java.lang.NullPointerException
   	at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectData.setField(ReflectData.java:139)
   	at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
   	at org.apache.pulsar.shade.org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:122)
   	at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:310)
   	at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
   	at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
   	at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
   	at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
   	at org.apache.pulsar.client.impl.schema.reader.AvroReader.read(AvroReader.java:56)
   	at org.apache.pulsar.client.api.schema.SchemaReader.read(SchemaReader.java:36)
   	at org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:98)
   ```
   
   An additional observation is that msg.getValue() does work only that it provides GenericAvroRecords which are not easily usable.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services