[GitHub] [pulsar] gaoran10 edited a comment on issue #7407: Support for tombstones (null value in messages) does not work

[GitBox <gi...@apache.org>]

gaoran10 edited a comment on issue #7407:
URL: https://github.com/apache/pulsar/issues/7407#issuecomment-698094497


   @fmiguelez Sorry, late response.
   
   1. `producer.newMessage().key("1").send()`
   
   Currently, if we send messages as above the message payload is `ByteBuffer.allocate(0)`, thus if the topic with schema the decoder can't decode an empty byte array, this will cause java.io.EOFException as below.
   
   ```
   org.apache.pulsar.client.api.SchemaSerializationException: java.io.EOFException
   
   	at org.apache.pulsar.client.impl.schema.reader.AvroReader.read(AvroReader.java:77)
   	at org.apache.pulsar.client.api.schema.SchemaReader.read(SchemaReader.java:36)
   	at org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:107)
   	at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:301)
   	at org.apache.pulsar.broker.transaction.TransactionProduceTest.test(TransactionProduceTest.java:639)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
   	at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
   	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
   	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
   	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
   	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
   	at org.testng.TestRunner.privateRun(TestRunner.java:648)
   	at org.testng.TestRunner.run(TestRunner.java:505)
   	at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
   	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
   	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
   	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
   	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
   	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
   	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
   	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
   	at org.testng.TestNG.runSuites(TestNG.java:1049)
   	at org.testng.TestNG.run(TestNG.java:1017)
   	at com.intellij.rt.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:66)
   	at com.intellij.rt.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:110)
   Caused by: java.io.EOFException
   	at org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:509)
   	at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:149)
   	at org.apache.avro.io.BinaryDecoder.readIndex(BinaryDecoder.java:460)
   	at org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:282)
   	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:178)
   	at org.apache.avro.reflect.ReflectDatumReader.readField(ReflectDatumReader.java:290)
   	at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:237)
   	at org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
   	at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:170)
   	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
   	at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)
   	at org.apache.pulsar.client.impl.schema.reader.AvroReader.read(AvroReader.java:75)
   	... 28 more
   ```
   
   2. `producer.newMessage().key("1").value(null).send()`
   
   If we specify the value is null, the messageMetaData will record the flag `msgMetadataBuilder.setNullValue(true)`, the consumer client will check the nullValue flag before schema decoder decode the payload, if the flag is true will return null directly.
   
   3. Currently, the message value has three forms, normal value, empty byte array, and null value, maybe we could return a null value if the schema message payload is empty byte array or null value, if the topic not use the schema consumer could return the original message payload (empty byte array or null value). please confirm @sijie 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org