You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/11/28 13:21:05 UTC
[GitHub] [pulsar] alpreu opened a new issue, #18658: [Bug] Records with BytesSchema are now sent as AUTO_CONSUME
alpreu opened a new issue, #18658:
URL: https://github.com/apache/pulsar/issues/18658
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
The affected version is Pulsar 2.9.3
### Minimal reproduce step
1) Produce a message without explicitly setting a schema:
```
Producer<byte[]> producer = client.newProducer().topic("triage-topic").create();
JsonObject json = new JsonObject();
json.addProperty("id", "aaa");
byte[] payload = json.toString().getBytes(StandardCharsets.UTF_8);
producer.sendAsync(payload);
```
2) Deploy a `Sink<GenericRecord>` connector (e.g. https://github.com/streamnative/pulsar-io-cloud-storage) consuming the topic above
3) Check the record schema
```
@Override
public void write(Record<GenericRecord> record) throws Exception {
log.info(record);
log.info(record.getSchema().toString());
}
```
### What did you expect to see?
Pulsar 2.9.2
```
SinkRecord(sourceRecord=PulsarRecord(topicName=Optional[persistent://public/default/triage-topic], partition=0, message=Optional[org.apache.pulsar.client.impl.MessageImpl@861a908], schema=org.apache.pulsar.client.impl.schema.BytesSchema@2a094ea1, failFunction=org.apache.pulsar.functions.source.PulsarSource$$Lambda$329/0x00000008006451e0@4fb29f58, ackFunction=org.apache.pulsar.functions.source.PulsarSource$$Lambda$328/0x0000000800644fb8@127a2fbb), value=[B@1756fd15)
org.apache.pulsar.client.impl.schema.BytesSchema@2a094ea1
```
### What did you see instead?
Pulsar 2.9.3
```
SinkRecord(sourceRecord=PulsarRecord(topicName=Optional[persistent://public/default/triage-topic], partition=0, message=Optional[org.apache.pulsar.client.impl.MessageImpl@2b63d4b1], schema=AUTO_CONSUME({schemaVersion=,schemaType=BYTES}), failFunction=org.apache.pulsar.functions.source.PulsarSource$$Lambda$330/0x0000000800643d90@7addff50, ackFunction=org.apache.pulsar.functions.source.PulsarSource$$Lambda$329/0x0000000800643b68@513acbbe), value=[B@1ed6b0ae)
AUTO_CONSUME({schemaVersion=,schemaType=BYTES})
```
### Anything else?
In Pulsar 2.10.1 the Schema is again `BytesSchema`, so the bug was only introduced in 2.9.3 and does not affect 2.10
Maybe it is related to https://github.com/apache/pulsar/pull/15622?
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #18658: [Bug] Records with BytesSchema are now sent as AUTO_CONSUME
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #18658:
URL: https://github.com/apache/pulsar/issues/18658#issuecomment-1330206036
I believe that Record.getSchema() should return BytesSchema, but maybe something is broken on 2.9.x.
@alpreu as a workaround you can try with `record.getMessage().getReaderSchema()` or something like that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on issue #18658: [Bug] Records with BytesSchema are now sent as AUTO_CONSUME
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on issue #18658:
URL: https://github.com/apache/pulsar/issues/18658#issuecomment-1330679994
And if you want to sink messages with Bytes schema, this [PR](https://github.com/streamnative/pulsar-io-cloud-storage/pull/508) may be helpful to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] alpreu commented on issue #18658: [Bug] Records with BytesSchema are now sent as AUTO_CONSUME
Posted by GitBox <gi...@apache.org>.
alpreu commented on issue #18658:
URL: https://github.com/apache/pulsar/issues/18658#issuecomment-1340592802
I'm closing this as the fixing commit has been backported to 2.9: https://github.com/apache/pulsar/commit/7c4187fb9c2b737a0a33637d5c360cae176d7c9b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] alpreu closed issue #18658: [Bug] Records with BytesSchema are now sent as AUTO_CONSUME
Posted by GitBox <gi...@apache.org>.
alpreu closed issue #18658: [Bug] Records with BytesSchema are now sent as AUTO_CONSUME
URL: https://github.com/apache/pulsar/issues/18658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on issue #18658: [Bug] Records with BytesSchema are now sent as AUTO_CONSUME
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on issue #18658:
URL: https://github.com/apache/pulsar/issues/18658#issuecomment-1330627581
The [PR15622](https://github.com/apache/pulsar/pull/15622) changes the logic of the method `getReaderSchema`.
https://github.com/apache/pulsar/blob/58ad3d09ddca46e9e2805e9c7a37c0de1a5c302d/pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageImpl.java#L386-L404
Before this PR, the AutoSchema will be parsed by this code, and if the schemaVersion = null, it will return BytesSchema.
>return Optional.of(((AutoConsumeSchema) schema).atSchemaVersion(schemaVersion));
After the PR, the AutoSchema without schemaVersion will be returned directly.
This is why in some 2.9.2.x the sink connector can receive Bytes schema but in 2.9.3.x only receives AutoConsumeSchema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #18658: [Bug] Records with BytesSchema are now sent as AUTO_CONSUME
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #18658:
URL: https://github.com/apache/pulsar/issues/18658#issuecomment-1330203638
@cbornet can you please take a look ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] liangyepianzhou commented on issue #18658: [Bug] Records with BytesSchema are now sent as AUTO_CONSUME
Posted by GitBox <gi...@apache.org>.
liangyepianzhou commented on issue #18658:
URL: https://github.com/apache/pulsar/issues/18658#issuecomment-1330638069
The [PR14847](https://github.com/apache/pulsar/pull/14847) adds a conversion here to ensure the record sent to the sink connector will not be an AutoConsume Scheme.
https://github.com/apache/pulsar/blob/58ad3d09ddca46e9e2805e9c7a37c0de1a5c302d/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java#L122
This is why in 2.10.x the sink connector regained the Bytes Schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org