You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/02/17 06:41:00 UTC
[GitHub] [pulsar] rvashishth opened a new issue #9602: Could not add AVRO schema using producer client
rvashishth opened a new issue #9602:
URL: https://github.com/apache/pulsar/issues/9602
**Describe the bug**
Unable to add AVRO schema on a topic using producer client.
**To Reproduce**
Steps to reproduce the behavior:
1. create a topic
2. produce a message with json schema
```
Producer<User> producer = client.newProducer(Schema.JSON(User.class)).topic(Commons.TOPIC).create();
User user = new User("rahul",31);
MessageId msgId = producer.send(user);
```
3. get schema api - return json schema type
```
{
"getSchemaResponses": [
{
"version": 0,
"type": "JSON",
"timestamp": 0,
"data": "{\"type\":\"record\",\"name\":\"User\",\"namespace\":\"com.optum.link.pulsar\",\"fields\":[{\"name\":\"name\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"age\",\"type\":\"int\"}]}",
"properties": {
"__jsr310ConversionEnabled": "false",
"__alwaysAllowNull": "true"
}
}
]
}
```
4. produce 2nd message with avro schema
```
Producer<User> producer = client.newProducer(Schema.AVRO(User.class)).topic(Commons.TOPIC).create();
User user = new User("josh",31);
MessageId msgId = producer.send(user);
```
5. get schema api - returns only previously created json schema object. it did not include AVRO schema
```
{
"getSchemaResponses": [
{
"version": 0,
"type": "JSON",
"timestamp": 0,
"data": "{\"type\":\"record\",\"name\":\"User\",\"namespace\":\"com.optum.link.pulsar\",\"fields\":[{\"name\":\"name\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"age\",\"type\":\"int\"}]}",
"properties": {
"__jsr310ConversionEnabled": "false",
"__alwaysAllowNull": "true"
}
}
]
}
```
**Expected behavior**
Avro schema should be added pulsar schema registry, or if this is not feasible then producer client should throw error
**Screenshots**
NA
**Desktop (please complete the following information):**
- OS: iOS
- Pulsar: 2.7.0
- Pulsar Client: 2.7.0
**Additional context**
I could successfully produce/consume messages using json schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-789000970
@ta1meng We are working on the PR to fix this problem. The difference between the AVRO schema and JSON schema is the serialization, JSON schema encode to the JSON bytes and the AVRO schema encode to the AVRO format bytes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 edited a comment on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
congbobo184 edited a comment on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-781140574
because we record the json schema by avro schema generate, broker validate the compatibility unified management through avro. the above example ```User.class``` Schema.JSON and Schema.AVRO generate the schema data are the same. So you can use the ```Schema.Avro(User.class)``` send message. This logic was like this before. This is a wrong logic, but if change this logic, it maybe produce the problem after broker version update when previous can connect producer or consumer will not connect correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ta1meng commented on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
ta1meng commented on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-786144750
If the validation logic will be fixed later, could you publish some documentation explaining the semantic difference between JSON schemas and AVRO schemas?
For example, does an AVRO schema support more field types than JSON schemas? Are there syntactical differences?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] ta1meng commented on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
ta1meng commented on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-784538699
I observed the same behavior using the Java client.
Was it really the wrong logic @congbobo184?
Because one interpretation is that the "JSON" schema type means the JSON-formatted Avro schema type, so the two types, "AVRO" and "JSON", should be interchangeable inside Pulsar?
Avro schemas can also be binary-encoded. However, I did not see support for binary Avro schemas in SchemaDefinitionBuilder inside SchemaDefinitionBuilder.java.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rvashishth commented on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
rvashishth commented on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-780359779
But still the producer client didn't throw any exception and produced message successfully.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] limingnihao commented on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
limingnihao commented on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-790369407
I will fix this issue, please assign it to me. A check for Scheme Type will be added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 edited a comment on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
congbobo184 edited a comment on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-781140574
because we record the json schema by avro schema generate, broker validate the compatibility unified management through avro. the above example ```User.class``` Schema.JSON and Schema.AVRO generate the schema data are the same. So you can use the ```Schema.Avro(User.class)``` send message. This logic was like this before. This is a wrong logic, but if change this logic, it maybe produce the problem after broker version update when previous can connect producer or consumer will not connect correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-781140574
because we record the json schema by avro schema generate, broker validate the compatibility unified management through avro. the above example ```User.class``` Schema.JSON and Schema.AVRO are the same. So you can use the ```Schema.Avro(User.class)``` send message. This logic was like this before. This is a wrong logic, but if change this logic, it maybe produce the problem after broker version update when previous can connect producer or consumer will not connect correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 edited a comment on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
congbobo184 edited a comment on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-784656957
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #9602:
URL: https://github.com/apache/pulsar/issues/9602
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] rvashishth edited a comment on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
rvashishth edited a comment on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-780359779
But still the producer client didn't throw any exception and produced message successfully.
Pulsar client might need a fix so it fails and propagate exception
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] congbobo184 commented on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
congbobo184 commented on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-784656957
now, schema check don't check the schema type, the avro logic will be fixed later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #9602: Could not add AVRO schema using producer client
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9602:
URL: https://github.com/apache/pulsar/issues/9602#issuecomment-780352270
@rvashishth This is expected behavior, you can get more details from http://pulsar.apache.org/docs/en/schema-evolution-compatibility. The JSON schema and AVRO schema are incompatible with each other. So the step 4 that you mentioned should fail with `schema incompatible exception`. If the new schema incompatible with the old schemas(based on the compatibility policy), the new schema will not update, so you got the old one (JSON schema) when you get schema info again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org