You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "jiangpengcheng (via GitHub)" <gi...@apache.org> on 2025/02/10 00:32:01 UTC
[PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
jiangpengcheng opened a new pull request, #23954:
URL: https://github.com/apache/pulsar/pull/23954
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see *[Guideline - Pulsar PR Naming Convention](https://pulsar.apache.org/contribute/develop-semantic-title/)*.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
-->
<!-- Either this PR fixes an issue, -->
### Motivation
The Kafka source connector support schema registered in schema registry, but it only support Avro for now, while the schema registry also supports Protobuf, it's better to support it too
### Modifications
Update the Kafka source connector to make it support Protobuf schema
### Verifying this change
- [x] Make sure that the change passes the CI checks.
- [x] This change added tests and can be verified as follows:
- *Added integration tests for using `KafkaProtobufDeserializer` to create Kafka source connector*
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: https://github.com/jiangpengcheng/pulsar/pull/39
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "jiangpengcheng (via GitHub)" <gi...@apache.org>.
jiangpengcheng commented on PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#issuecomment-2719505994
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#discussion_r2056471225
##########
pulsar-io/kafka/pom.xml:
##########
@@ -104,6 +104,12 @@
<version>${confluent.version}</version>
</dependency>
+ <dependency>
+ <groupId>io.confluent</groupId>
+ <artifactId>kafka-protobuf-serializer</artifactId>
Review Comment:
I found an existing issue about the license for protobuf libraries: https://github.com/confluentinc/schema-registry/issues/1558
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#issuecomment-2824831442
One potential problem is the Confluent schema registry licenses. It says
> The project is licensed under the [Confluent Community License](https://github.com/confluentinc/schema-registry/blob/master/LICENSE-ConfluentCommunity), except for the client-* and avro-* libs, which are under the [Apache 2.0 license](https://github.com/confluentinc/schema-registry/blob/master/LICENSE-Apache). See LICENSE file in each subfolder for detailed license agreement.
for example, https://github.com/confluentinc/schema-registry/blob/master/protobuf-serializer/src/main/java/io/confluent/kafka/serializers/protobuf/KafkaProtobufDeserializer.java doesn't have Apache 2.0 license.
https://github.com/confluentinc/schema-registry/blob/master/avro-serializer/src/main/java/io/confluent/kafka/serializers/KafkaAvroDeserializer.java is under Apache 2.0 license.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#discussion_r2056453888
##########
pulsar-io/kafka/pom.xml:
##########
@@ -104,6 +104,12 @@
<version>${confluent.version}</version>
</dependency>
+ <dependency>
+ <groupId>io.confluent</groupId>
+ <artifactId>kafka-protobuf-serializer</artifactId>
Review Comment:
In the pom file https://packages.confluent.io/maven/io/confluent/kafka-protobuf-serializer/7.8.2/kafka-protobuf-serializer-7.8.2.pom, there's Apache 2.0 license. This is also the case for 6.2.8 version, https://packages.confluent.io/maven/io/confluent/kafka-protobuf-serializer/6.2.8/kafka-protobuf-serializer-6.2.8.pom . I wonder if they just have forgotten to make things consistent in the repository.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "jiangpengcheng (via GitHub)" <gi...@apache.org>.
jiangpengcheng closed pull request #23954: [improve][io] Support Protobuf schema for Kafka source connector
URL: https://github.com/apache/pulsar/pull/23954
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#issuecomment-2719548788
## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/23954?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `36.70886%` with `50 lines` in your changes missing coverage. Please review.
> Project coverage is 74.20%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`ba65b6b`)](https://app.codecov.io/gh/apache/pulsar/commit/ba65b6b0f3738d79be167f45eb75c1cb3ff2589a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 957 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/23954?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...a/org/apache/pulsar/io/kafka/KafkaSchemaCache.java](https://app.codecov.io/gh/apache/pulsar/pull/23954?src=pr&el=tree&filepath=pulsar-io%2Fkafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fio%2Fkafka%2FKafkaSchemaCache.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWlvL2thZmthL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvaW8va2Fma2EvS2Fma2FTY2hlbWFDYWNoZS5qYXZh) | 16.32% | [41 Missing :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/23954?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...a/org/apache/pulsar/io/kafka/KafkaBytesSource.java](https://app.codecov.io/gh/apache/pulsar/pull/23954?src=pr&el=tree&filepath=pulsar-io%2Fkafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fio%2Fkafka%2FKafkaBytesSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWlvL2thZmthL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvaW8va2Fma2EvS2Fma2FCeXRlc1NvdXJjZS5qYXZh) | 70.00% | [6 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/23954?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
[](https://app.codecov.io/gh/apache/pulsar/pull/23954?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
```diff
@@ Coverage Diff @@
## master #23954 +/- ##
============================================
+ Coverage 73.57% 74.20% +0.63%
+ Complexity 32624 32438 -186
============================================
Files 1877 1863 -14
Lines 139502 144332 +4830
Branches 15299 16467 +1168
============================================
+ Hits 102638 107104 +4466
+ Misses 28908 28771 -137
- Partials 7956 8457 +501
```
| [Flag](https://app.codecov.io/gh/apache/pulsar/pull/23954/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ | |
|---|---|---|
| [inttests](https://app.codecov.io/gh/apache/pulsar/pull/23954/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `26.74% <0.00%> (+2.16%)` | :arrow_up: |
| [systests](https://app.codecov.io/gh/apache/pulsar/pull/23954/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `23.16% <0.00%> (-1.16%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pulsar/pull/23954/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.72% <36.70%> (+0.87%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/pulsar/pull/23954?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Ξ | |
|---|---|---|
| [...a/org/apache/pulsar/io/kafka/KafkaBytesSource.java](https://app.codecov.io/gh/apache/pulsar/pull/23954?src=pr&el=tree&filepath=pulsar-io%2Fkafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fio%2Fkafka%2FKafkaBytesSource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWlvL2thZmthL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvaW8va2Fma2EvS2Fma2FCeXRlc1NvdXJjZS5qYXZh) | `71.96% <70.00%> (+1.50%)` | :arrow_up: |
| [...a/org/apache/pulsar/io/kafka/KafkaSchemaCache.java](https://app.codecov.io/gh/apache/pulsar/pull/23954?src=pr&el=tree&filepath=pulsar-io%2Fkafka%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fio%2Fkafka%2FKafkaSchemaCache.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWlvL2thZmthL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvaW8va2Fma2EvS2Fma2FTY2hlbWFDYWNoZS5qYXZh) | `16.32% <16.32%> (ΓΈ)` | |
... and [1064 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/23954/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary>π New features to boost your workflow: </summary>
- β [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- π¦ [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#issuecomment-2824733350
@jiangpengcheng I have resolved the merge conflict after #24201 changes which upgraded Confluent Platform version to 7.8.2 and Kafka client version to 3.8.1. The integration test seems to fail now. I'm not sure exactly what the problem is.
I guess you are already aware that the way to run the integration test locally is to first build the docker image with this command:
```
mvn -B -am -pl distribution/io,distribution/offloaders,distribution/server,distribution/shell,tests/docker-images/latest-version-image install \
-Pmain,docker -Dmaven.test.skip=true \
-Dspotbugs.skip=true -Dlicense.skip=true -Dcheckstyle.skip=true -Drat.skip=true
```
And then run the test
```
mvn -DredirectTestOutputToFile=false -DtestRetryCount=0 test -DintegrationTests -pl tests/integration -Dtest=KafkaSourceTest#testProtobuf
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#issuecomment-2829664162
I also found Apicurio Schema registry libs which are Apache 2.0 licensed, https://github.com/Apicurio/apicurio-registry/tree/main/serdes/kafka/protobuf-serde/src/main/java/io/apicurio/registry/serde/protobuf . However, it looks like the wire format is Apicurio specific and not compatible with Confluent Schema Registry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "jiangpengcheng (via GitHub)" <gi...@apache.org>.
jiangpengcheng commented on code in PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#discussion_r2057994879
##########
pulsar-io/kafka/pom.xml:
##########
@@ -104,6 +104,12 @@
<version>${confluent.version}</version>
</dependency>
+ <dependency>
+ <groupId>io.confluent</groupId>
+ <artifactId>kafka-protobuf-serializer</artifactId>
Review Comment:
oh, I didn't check the library license, will close this then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#discussion_r2055392879
##########
pulsar-io/kafka/pom.xml:
##########
@@ -104,6 +104,12 @@
<version>${confluent.version}</version>
</dependency>
+ <dependency>
+ <groupId>io.confluent</groupId>
+ <artifactId>kafka-protobuf-serializer</artifactId>
+ <version>${confluent.version}</version>
Review Comment:
I think it's time to bump `confluent.version`. It's currently `6.2.8` which was released on Dec 16, 2022. [Available versions](https://mvnrepository.com/artifact/io.confluent/kafka-protobuf-serializer).
I have created a separate PR to upgrade confluent platform version and kafka client version, #24201.
Since it addresses CVEs, I believe that it would be useful to first merge it and then rebase this PR after that to ensure that changes are compatible with recent versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "jiangpengcheng (via GitHub)" <gi...@apache.org>.
jiangpengcheng commented on PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#issuecomment-2749936057
Hi @lhotari, could u help to review this pr, thx~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#discussion_r2056443593
##########
pulsar-io/kafka/pom.xml:
##########
@@ -104,6 +104,12 @@
<version>${confluent.version}</version>
</dependency>
+ <dependency>
+ <groupId>io.confluent</groupId>
+ <artifactId>kafka-protobuf-serializer</artifactId>
Review Comment:
According to the source repository, https://github.com/confluentinc/schema-registry, this library has a license of Confluent Community License. This didn't change between the previous
https://github.com/confluentinc/schema-registry/tree/v6.2.8 version and the currently used https://github.com/confluentinc/schema-registry/tree/v7.8.2 version.
The information provided by Confluent about the license is slightly conflicting since it says that client libraries are with Apache 2.0 license, but at the same time, it says "See LICENSE file in each subfolder for detailed license agreement". However, the license headers for protobuf classes are very explicit about the license. For Avro related classes, there's an Apache 2.0 license header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#issuecomment-2829643854
It looks like AWS Glue Schema Registry Library contains the classes that we'd need for parsing the Confluent Schema Registry protobuf [wire format](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#messages-wire-format).
Under https://github.com/awslabs/aws-glue-schema-registry/tree/master/serializer-deserializer/src/main/java/com/amazonaws/services/schemaregistry there are both serializers and deserializers which we could use as a replacement for the Confluent Community Licensed libraries. We could continue to use the Apache 2.0 licensed Confluent Schema Registry libraries.
There are limitations in the AWS Glue libs, such as not supporting imported schemas like Confluent Schema Registry supports: https://github.com/awslabs/aws-glue-schema-registry/blob/72cbca0b05a758f0a775c39e580a15e7f19613fb/serializer-deserializer/src/main/java/com/amazonaws/services/schemaregistry/serializers/protobuf/MessageIndexFinder.java#L71-L72
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#discussion_r2056379622
##########
pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaBytesSource.java:
##########
@@ -230,25 +271,31 @@ public BytesWithKafkaSchema deserialize(String topic, byte[] payload) {
ByteBuffer buffer = ByteBuffer.wrap(payload);
buffer.get(); // magic number
int id = buffer.getInt();
- return new BytesWithKafkaSchema(buffer, id);
+ // the kafka protobuf serializer encodes the MessageIndexes in the payload, we need to skip them
+ if (schemaType == SchemaType.PROTOBUF_NATIVE) {
+ MessageIndexes.readFrom(buffer);
+ }
Review Comment:
[wire format docs](https://docs.confluent.io/platform/current/schema-registry/fundamentals/serdes-develop/index.html#messages-wire-format)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#discussion_r2056464981
##########
pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaBytesSource.java:
##########
@@ -230,25 +271,31 @@ public BytesWithKafkaSchema deserialize(String topic, byte[] payload) {
ByteBuffer buffer = ByteBuffer.wrap(payload);
buffer.get(); // magic number
int id = buffer.getInt();
- return new BytesWithKafkaSchema(buffer, id);
+ // the kafka protobuf serializer encodes the MessageIndexes in the payload, we need to skip them
+ if (schemaType == SchemaType.PROTOBUF_NATIVE) {
+ MessageIndexes.readFrom(buffer);
+ }
Review Comment:
Once the license issue is resolved, we could come back to this detail. If the code for KafkaProtobufDeserializer would be Apache 2.0 licensed, we could rather safely look at the code and see how the proper schema resolution can be handled for protobuf encoded messages. As long as we have the license issue, we better not copy-paste code due to IPR violation risk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#discussion_r2056364150
##########
pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaBytesSource.java:
##########
@@ -230,25 +271,31 @@ public BytesWithKafkaSchema deserialize(String topic, byte[] payload) {
ByteBuffer buffer = ByteBuffer.wrap(payload);
buffer.get(); // magic number
int id = buffer.getInt();
- return new BytesWithKafkaSchema(buffer, id);
+ // the kafka protobuf serializer encodes the MessageIndexes in the payload, we need to skip them
+ if (schemaType == SchemaType.PROTOBUF_NATIVE) {
+ MessageIndexes.readFrom(buffer);
+ }
Review Comment:
There seems to be a meaning why the `MessageIndexes` exists. Each protobuf schema can include multiple message definitions and the `MessageIndexes` contains a solution for referencing a specific message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#issuecomment-2829687115
More OSS schema registry clients with protobuf support:
* https://github.com/nryanov/schemakeeper
* https://github.com/raystack/stencil/tree/main/clients/java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#discussion_r2056484699
##########
pulsar-io/kafka/pom.xml:
##########
@@ -104,6 +104,12 @@
<version>${confluent.version}</version>
</dependency>
+ <dependency>
+ <groupId>io.confluent</groupId>
+ <artifactId>kafka-protobuf-serializer</artifactId>
Review Comment:
Confluent employee replied in 2021: https://github.com/confluentinc/schema-registry/issues/1558#issuecomment-956442463
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#discussion_r2056479905
##########
pulsar-io/kafka/pom.xml:
##########
@@ -104,6 +104,12 @@
<version>${confluent.version}</version>
</dependency>
+ <dependency>
+ <groupId>io.confluent</groupId>
+ <artifactId>kafka-protobuf-serializer</artifactId>
Review Comment:
there's even a branch to fix the issue dating back to 2021, but it was never merged: https://github.com/confluentinc/schema-registry/compare/protobuf-licensing
##########
pulsar-io/kafka/pom.xml:
##########
@@ -104,6 +104,12 @@
<version>${confluent.version}</version>
</dependency>
+ <dependency>
+ <groupId>io.confluent</groupId>
+ <artifactId>kafka-protobuf-serializer</artifactId>
Review Comment:
there's even a branch to fix the issue, dating back to 2021, but it was never merged: https://github.com/confluentinc/schema-registry/compare/protobuf-licensing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][io] Support Protobuf schema for Kafka source connector [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #23954:
URL: https://github.com/apache/pulsar/pull/23954#issuecomment-2826624410
We cannot add a dependency on the `io.confluent:kafka-protobuf-serializer` library since it is under the Confluent Community License, as [briefly commented in an issue discussion](https://github.com/confluentinc/schema-registry/issues/1558#issuecomment-956442463).
This is how Claude AI explained the licensing issue:
> The Confluent Community License (CCL) and Apache 2.0 license have important compatibility considerations:
>
> The main challenge is that the Confluent Community License is not an open source license according to the Open Source Initiative's definition. It contains field-of-use restrictions that prevent using the software to build competing services.
>
> Specifically:
> - Apache 2.0 is a permissive open source license that allows code to be used in proprietary software
> - The CCL restricts using the licensed software to create "Competitive Offerings" (competing SaaS products)
>
> This creates asymmetric compatibility:
> - You can include Apache 2.0 code in a CCL-licensed project
> - You generally cannot include CCL-licensed code in an Apache 2.0 project without the resulting work becoming subject to CCL restrictions
>
> If you're developing software that combines components with these licenses, you should:
> 1. Carefully review the specific license texts
> 2. Consider the direction of code sharing between components
> 3. Consult with legal counsel for your specific use case
>
> If you need to maintain Apache 2.0 compliance, you'll typically need to avoid incorporating CCL-licensed code in your Apache project.
Unless `io.confluent:kafka-protobuf-serializer` gets relicensed under Apache 2.0 license, we cannot use it in Apache Pulsar. We should also avoid looking at the code to avoid any IPR issues and instead implement a "clean room" implementation from scratch using the documentation of the wire format, since one could assume that there's no IPR on the wire format itself.
We could also consider supporting protobuf encoded messages that aren't encoded using Confluent's schema registry client libraries and don't use the schema registry for dynamically retrieving the schema. In those cases, the schema would have to be provided in the source connector config. That would be a different feature than what this PR implements.
@jiangpengcheng WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org