You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/04/08 15:05:05 UTC
[GitHub] [pulsar] shiv4289 opened a new issue #6694: [AVRO SCHEMA] Reset
cursor on stream if avro encode fails
shiv4289 opened a new issue #6694: [AVRO SCHEMA] Reset cursor on stream if avro encode fails
URL: https://github.com/apache/pulsar/issues/6694
**Describe the bug**
If the avro encode fails, the cursor in the stream is not reset. For the next message the positions are thus arbitrary.
**To Reproduce**
Steps to reproduce the behavior:
1. Add an avro schema (Add it using the json string representing the avro schema). Keep the second field not null
2. Write a message that sends null for the second field. This means first field was written into the buffer and the avro encode crashed before second field could be written.
3. Write the correct message and notice that the correct message is not written from position 0 in the buffer.
4. The decode shows that fields from first and second messages are mixed up and there are multiple different errors on decode. Below is a sample stacktrace. on the consumer side when msg.getValue() is invoked.
```
Exception in thread "main" org.apache.pulsar.shade.org.apache.avro.AvroRuntimeException: Malformed data. Length is negative: -52
at org.apache.pulsar.shade.org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336)
at org.apache.pulsar.shade.org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263)
at org.apache.pulsar.shade.org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:414)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:181)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
at org.apache.pulsar.client.impl.schema.generic.GenericAvroReader.read(GenericAvroReader.java:75)
at org.apache.pulsar.client.impl.schema.generic.GenericAvroReader.read(GenericAvroReader.java:42)
at org.apache.pulsar.client.api.schema.SchemaReader.read(SchemaReader.java:36)
at org.apache.pulsar.client.impl.schema.StructSchema.decode(StructSchema.java:98)
at org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:96)
at org.apache.pulsar.client.impl.schema.AutoConsumeSchema.decode(AutoConsumeSchema.java:39)
at org.apache.pulsar.client.impl.MessageImpl.getValue(MessageImpl.java:275)
```
The bugs was noticed in 2.4, 2.5 as well as the latest
**Expected behavior**
The first message should be discarded. The second message should be produced and consumed correctly.
**Additional context**
The debugger reveals the issue when [this](https://github.com/apache/pulsar/blob/21f6dcdda406423f1bcfe4761831b7297d659fd1/pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/writer/AvroWriter.java#L44) line crashes. The following flush() that resets the cursor to 0 is not executed and thence the issue. Copying code from the link for reference:
```
public synchronized byte[] write(T message) {
try {
writer.write(message, this.encoder); // if crash here, no flush so no reset cursor to 0.
this.encoder.flush();
return this.byteArrayOutputStream.toByteArray();
} catch (Exception e) {
throw new SchemaSerializationException(e);
} finally {
this.byteArrayOutputStream.reset();
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [pulsar] sijie closed issue #6694: [AVRO SCHEMA] Reset cursor on
stream if avro encode fails
Posted by GitBox <gi...@apache.org>.
sijie closed issue #6694: [AVRO SCHEMA] Reset cursor on stream if avro encode fails
URL: https://github.com/apache/pulsar/issues/6694
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services