You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/06/02 03:23:03 UTC
[GitHub] [pulsar] momo-jun opened a new pull request, #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
momo-jun opened a new pull request, #15887:
URL: https://github.com/apache/pulsar/pull/15887
Fixes #15855
### Modifications
1. Add code example and more text for `schemaRegistryCompatibilityCheckers`.
2. Refactor the section.
The preview looks good.
<img width="1409" alt="image" src="https://user-images.githubusercontent.com/60642177/171546070-a914dc40-e089-4613-a100-da28478ad7d9.png">
### Documentation
- [ ] `doc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#issuecomment-1151857610
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#issuecomment-1144393803
@momo-jun:Thanks for providing doc info!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
momo-jun commented on PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#issuecomment-1149874267
Ping @Anonymitaet for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#discussion_r894188628
##########
conf/broker.conf:
##########
@@ -557,6 +557,9 @@ zookeeperSessionExpiredPolicy=reconnect
# Enable or disable system topic
systemTopicEnabled=true
+# Deploy the schema compatibility checker for a specific schema type to enforce schema compatibility check
+schemaRegistryCompatibilityCheckers=org.apache.pulsar.broker.service.schema.JsonSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.AvroSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaCompatibilityCheck
Review Comment:
The test has been completed successfully :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on code in PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#discussion_r893027775
##########
site2/docs/schema-evolution-compatibility.md:
##########
@@ -30,19 +30,26 @@ For more information, see [Schema compatibility check strategy](#schema-compatib
### How does Pulsar support schema evolution?
-1. When a producer/consumer/reader connects to a broker, the broker deploys the schema compatibility checker configured by `schemaRegistryCompatibilityCheckers` to enforce schema compatibility check.
+The process of how Pulsar supports schema evolution is described as follows.
- The schema compatibility checker is one instance per schema type.
+1. The producer/consumer/reader sends the `SchemaInfo` of its client to brokers.
- Currently, Avro and JSON have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables schema evolution.
-
-2. The producer/consumer/reader sends its client `SchemaInfo` to the broker.
+2. Brokers recognize the schema type and deploy the schema compatibility checker `schemaRegistryCompatibilityCheckers` for that schema type to enforce the schema compatibility check. By default, the value of `schemaRegistryCompatibilityCheckers` in the `conf/broker.conf` or `conf/standalone.conf` file is as follows.
-3. The broker knows the schema type and locates the schema compatibility checker for that type.
+ ```properties
+ schemaRegistryCompatibilityCheckers=org.apache.pulsar.broker.service.schema.JsonSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.AvroSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaCompatibilityCheck
+ ```
+
+ :::note
+
+ Each schema type corresponds to one instance of schema compatibility checker. Currently, Avro, JSON and Protobuf have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables the schema evolution. In a word, schema evolution is only available in Avro, JSON and Protobuf schema.
Review Comment:
```suggestion
Each schema type corresponds to one instance of schema compatibility checker. Currently, Avro, JSON, and Protobuf have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables the schema evolution. In a word, schema evolution is only available in Avro, JSON, and Protobuf schema.
```
##########
site2/docs/schema-evolution-compatibility.md:
##########
@@ -30,19 +30,26 @@ For more information, see [Schema compatibility check strategy](#schema-compatib
### How does Pulsar support schema evolution?
-1. When a producer/consumer/reader connects to a broker, the broker deploys the schema compatibility checker configured by `schemaRegistryCompatibilityCheckers` to enforce schema compatibility check.
+The process of how Pulsar supports schema evolution is described as follows.
- The schema compatibility checker is one instance per schema type.
+1. The producer/consumer/reader sends the `SchemaInfo` of its client to brokers.
- Currently, Avro and JSON have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables schema evolution.
-
-2. The producer/consumer/reader sends its client `SchemaInfo` to the broker.
+2. Brokers recognize the schema type and deploy the schema compatibility checker `schemaRegistryCompatibilityCheckers` for that schema type to enforce the schema compatibility check. By default, the value of `schemaRegistryCompatibilityCheckers` in the `conf/broker.conf` or `conf/standalone.conf` file is as follows.
-3. The broker knows the schema type and locates the schema compatibility checker for that type.
+ ```properties
+ schemaRegistryCompatibilityCheckers=org.apache.pulsar.broker.service.schema.JsonSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.AvroSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaCompatibilityCheck
+ ```
+
+ :::note
+
+ Each schema type corresponds to one instance of schema compatibility checker. Currently, Avro, JSON and Protobuf have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables the schema evolution. In a word, schema evolution is only available in Avro, JSON and Protobuf schema.
Review Comment:
Use commas to separate items in a series of three or more. Use a comma before the conjunction that precedes the final item. https://docs.google.com/document/d/1lc5j4RtuLIzlEYCBo97AC8-U_3Erzs_lxpkDuseU0n4/edit#bookmark=id.b82f2ay5cpsc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece commented on a diff in pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
nodece commented on code in PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#discussion_r887622564
##########
site2/docs/schema-evolution-compatibility.md:
##########
@@ -30,19 +30,34 @@ For more information, see [Schema compatibility check strategy](#schema-compatib
### How does Pulsar support schema evolution?
-1. When a producer/consumer/reader connects to a broker, the broker deploys the schema compatibility checker configured by `schemaRegistryCompatibilityCheckers` to enforce schema compatibility check.
+The process of how Pulsar supports schema evolution is described as follows.
- The schema compatibility checker is one instance per schema type.
+1. The producer/consumer/reader sends its client `SchemaInfo` to brokers.
- Currently, Avro and JSON have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables schema evolution.
+2. Brokers recognize the schema type and deploy the schema compatibility checker `schemaRegistryCompatibilityCheckers` for that schema type to enforce schema compatibility check.
-2. The producer/consumer/reader sends its client `SchemaInfo` to the broker.
-
-3. The broker knows the schema type and locates the schema compatibility checker for that type.
+3. Brokers use the schema compatibility checker to check if the `SchemaInfo` is compatible with the latest schema of the topic by applying its compatibility check strategy. Currently, the compatibility check strategy is configured at the namespace level and applied to all the topics within that namespace.
+
+:::tip
+
+Each schema type corresponds to one instance of schema compatibility checker. Currently, Avro, JSON and Protobuff have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables schema evolution.
+
+:::
+
+The following code snippet describes the implementation of `schemaRegistryCompatibilityCheckers`. For more details, see [code](https://github.com/apache/pulsar/blob/bf194b557c48e2d3246e44f1fc28876932d8ecb8/pul[…]rc/main/java/org/apache/pulsar/broker/ServiceConfiguration.java).
Review Comment:
If you add schemaRegistryCompatibilityCheckers config above, you can remove this part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#issuecomment-1151003624
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet commented on a diff in pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
Anonymitaet commented on code in PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#discussion_r894108155
##########
conf/broker.conf:
##########
@@ -557,6 +557,9 @@ zookeeperSessionExpiredPolicy=reconnect
# Enable or disable system topic
systemTopicEnabled=true
+# Deploy the schema compatibility checker for a specific schema type to enforce schema compatibility check
+schemaRegistryCompatibilityCheckers=org.apache.pulsar.broker.service.schema.JsonSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.AvroSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaCompatibilityCheck
Review Comment:
I re-run the check again just now.
I checked the failure check yesterday and guess it might be caused by the overlength.
As we discussed w/ engineers just now, can you try to split into multiple lines if the check fails again?
Ignore this comment and ping me to merge if the check passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#issuecomment-1144393129
@momo-jun:Thanks for your contribution. For this PR, do we need to update docs?
(The [PR template contains info about doc](https://github.com/apache/pulsar/blob/master/.github/PULL_REQUEST_TEMPLATE.md#documentation), which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece commented on a diff in pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
nodece commented on code in PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#discussion_r887625650
##########
site2/docs/schema-evolution-compatibility.md:
##########
@@ -30,19 +30,34 @@ For more information, see [Schema compatibility check strategy](#schema-compatib
### How does Pulsar support schema evolution?
-1. When a producer/consumer/reader connects to a broker, the broker deploys the schema compatibility checker configured by `schemaRegistryCompatibilityCheckers` to enforce schema compatibility check.
+The process of how Pulsar supports schema evolution is described as follows.
- The schema compatibility checker is one instance per schema type.
+1. The producer/consumer/reader sends its client `SchemaInfo` to brokers.
- Currently, Avro and JSON have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables schema evolution.
+2. Brokers recognize the schema type and deploy the schema compatibility checker `schemaRegistryCompatibilityCheckers` for that schema type to enforce schema compatibility check.
Review Comment:
You also need to add `schemaRegistryCompatibilityCheckers` to https://pulsar.apache.org/docs/next/reference-configuration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#discussion_r893138409
##########
site2/docs/schema-evolution-compatibility.md:
##########
@@ -30,19 +30,26 @@ For more information, see [Schema compatibility check strategy](#schema-compatib
### How does Pulsar support schema evolution?
-1. When a producer/consumer/reader connects to a broker, the broker deploys the schema compatibility checker configured by `schemaRegistryCompatibilityCheckers` to enforce schema compatibility check.
+The process of how Pulsar supports schema evolution is described as follows.
- The schema compatibility checker is one instance per schema type.
+1. The producer/consumer/reader sends the `SchemaInfo` of its client to brokers.
- Currently, Avro and JSON have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables schema evolution.
-
-2. The producer/consumer/reader sends its client `SchemaInfo` to the broker.
+2. Brokers recognize the schema type and deploy the schema compatibility checker `schemaRegistryCompatibilityCheckers` for that schema type to enforce the schema compatibility check. By default, the value of `schemaRegistryCompatibilityCheckers` in the `conf/broker.conf` or `conf/standalone.conf` file is as follows.
-3. The broker knows the schema type and locates the schema compatibility checker for that type.
+ ```properties
+ schemaRegistryCompatibilityCheckers=org.apache.pulsar.broker.service.schema.JsonSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.AvroSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaCompatibilityCheck
+ ```
+
+ :::note
+
+ Each schema type corresponds to one instance of schema compatibility checker. Currently, Avro, JSON and Protobuf have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables the schema evolution. In a word, schema evolution is only available in Avro, JSON and Protobuf schema.
Review Comment:
Updated. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Anonymitaet merged pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
Anonymitaet merged PR #15887:
URL: https://github.com/apache/pulsar/pull/15887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] nodece commented on a diff in pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
nodece commented on code in PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#discussion_r887621175
##########
site2/docs/schema-evolution-compatibility.md:
##########
@@ -30,19 +30,34 @@ For more information, see [Schema compatibility check strategy](#schema-compatib
### How does Pulsar support schema evolution?
-1. When a producer/consumer/reader connects to a broker, the broker deploys the schema compatibility checker configured by `schemaRegistryCompatibilityCheckers` to enforce schema compatibility check.
+The process of how Pulsar supports schema evolution is described as follows.
- The schema compatibility checker is one instance per schema type.
+1. The producer/consumer/reader sends its client `SchemaInfo` to brokers.
- Currently, Avro and JSON have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables schema evolution.
+2. Brokers recognize the schema type and deploy the schema compatibility checker `schemaRegistryCompatibilityCheckers` for that schema type to enforce schema compatibility check.
Review Comment:
I suggest adding `schemaRegistryCompatibilityCheckers` config here, so like:
```
schemaRegistryCompatibilityCheckers=org.apache.pulsar.broker.service.schema.JsonSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.AvroSchemaCompatibilityCheck,org.apache.pulsar.broker.service.schema.ProtobufNativeSchemaCompatibilityCheck
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on a diff in pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
momo-jun commented on code in PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#discussion_r887747618
##########
site2/docs/schema-evolution-compatibility.md:
##########
@@ -30,19 +30,34 @@ For more information, see [Schema compatibility check strategy](#schema-compatib
### How does Pulsar support schema evolution?
-1. When a producer/consumer/reader connects to a broker, the broker deploys the schema compatibility checker configured by `schemaRegistryCompatibilityCheckers` to enforce schema compatibility check.
+The process of how Pulsar supports schema evolution is described as follows.
- The schema compatibility checker is one instance per schema type.
+1. The producer/consumer/reader sends its client `SchemaInfo` to brokers.
- Currently, Avro and JSON have their own compatibility checkers, while all the other schema types share the default compatibility checker which disables schema evolution.
+2. Brokers recognize the schema type and deploy the schema compatibility checker `schemaRegistryCompatibilityCheckers` for that schema type to enforce schema compatibility check.
Review Comment:
Need to double-check the default compatibility checker for other schema types. Seems it's not shown in the default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] momo-jun commented on pull request #15887: [fix][doc] Add context for schemaRegistryCompatibilityCheckers
Posted by GitBox <gi...@apache.org>.
momo-jun commented on PR #15887:
URL: https://github.com/apache/pulsar/pull/15887#issuecomment-1144392978
Ping @nodece for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org