[GitHub] [pulsar] fmiguelez commented on issue #4803: Pulsar schema doesn't support null value

[GitBox <gi...@apache.org>]

fmiguelez commented on issue #4803:
URL: https://github.com/apache/pulsar/issues/4803#issuecomment-625317625


   There is a workaround with direct consumer/reader interface since you can always verify if data is empty before calling getValue():
   
   Message<GenericRecord> lastMsg = reader.readNext(); lastEntity = lastMsg.getData().length > 0 ? lastMsg.getValue() : null;
   
   However there is not such option when working with functions. E.g. A Sink<GenericRecord> implementation will fail before calling write(Record<GenericRecord> record) if a message with no payload (valid tombstone) is received with this error:
   
   10:02:27.911 [dbus/test/rtsink-0] ERROR org.apache.pulsar.functions.instance.JavaInstanceRunnable - [dbus/test/rtsink:0] Uncaught exception in Java Instance
   org.apache.pulsar.client.api.SchemaSerializationException: java.io.EOFException
   at org.apache.pulsar.client.impl.schema.generic.GenericAvroReader.read(GenericAvroReader.java:80) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1]
   at org.apache.pulsar.client.impl.schema.generic.GenericAvroReader.read(GenericAvroReader.java:42) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1]
   at org.apache.pulsar.client.api.schema.SchemaReader.read(SchemaReader.java:36) ~[java-instance.jar:?]
   at org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:100) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1]
   at org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:96) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1]
   at org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:39) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1]
   at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:276) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1]
   at org.apache.pulsar.client.impl.TopicMessageImpl.getValue(TopicMessageImpl.java:143) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1]
   at org.apache.pulsar.functions.source.PulsarRecord.getValue(PulsarRecord.java:74) ~[org.apache.pulsar-pulsar-functions-instance-2.5.1.jar:2.5.1]
   at org.apache.pulsar.functions.instance.JavaInstanceRunnable.readInput(JavaInstanceRunnable.java:472) ~[org.apache.pulsar-pulsar-functions-instance-2.5.1.jar:?]
   at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:246) [org.apache.pulsar-pulsar-functions-instance-2.5.1.jar:?]
   at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
   Caused by: java.io.EOFException
   at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:509) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.io.BinaryDecoder.readLong(BinaryDecoder.java:181) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:279) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:209) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:467) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:458) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:182) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:248) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:237) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:170) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144) ~[org.apache.avro-avro-1.9.1.jar:1.9.1]
   at org.apache.pulsar.client.impl.schema.generic.GenericAvroReader.read(GenericAvroReader.java:75) ~[org.apache.pulsar-pulsar-client-original-2.5.1.jar:2.5.1]
   ... 11 more
   10:02:27.943 [dbus/test/rtsink-0] INFO org.apache.pulsar.functions.instance.JavaInstanceRunnable - Closing instance
   10:02:27.964 [dbus/test/rtsink-0] INFO org.apache.pulsar.client.impl.ConsumerImpl - [persistent://dbus/test/dummy-objects] [dbus/test/rtsink] Closed consumer
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org