You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/08/24 23:47:07 UTC
[GitHub] [kafka] jthompson6 opened a new pull request #9216: KAFKA-10428: Fix schema for header conversion in MirrorSourceTask.
jthompson6 opened a new pull request #9216:
URL: https://github.com/apache/kafka/pull/9216
The addBytes method adds the header using Schema.BYTES, which results in base64 encoding when the record is stored. SimpleHeaderConverter#toConnectHeader implements schema inference which we can use here, as the ConsumerRecord does not have header schema information.
Testing: I added schema verification to the existing MirrorSourceTaskTest#testSerde, with a string and int example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ryannedolan commented on pull request #9216: KAFKA-10428: Fix schema for header conversion in MirrorSourceTask.
Posted by GitBox <gi...@apache.org>.
ryannedolan commented on pull request #9216:
URL: https://github.com/apache/kafka/pull/9216#issuecomment-681062141
Thanks for finding this. Would it make sense to implement a new nop converter for this purpose? Not sure whether a KIP would be required -- maybe it can be private to the mirror package?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jthompson6 commented on pull request #9216: KAFKA-10428: Fix schema for header conversion in MirrorSourceTask.
Posted by GitBox <gi...@apache.org>.
jthompson6 commented on pull request #9216:
URL: https://github.com/apache/kafka/pull/9216#issuecomment-686701420
Sorry for the delay, I just tested adding `"header.converter": "org.apache.kafka.connect.converters.ByteArrayConverter"` to the connector config and it works.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ryannedolan commented on pull request #9216: KAFKA-10428: Fix schema for header conversion in MirrorSourceTask.
Posted by GitBox <gi...@apache.org>.
ryannedolan commented on pull request #9216:
URL: https://github.com/apache/kafka/pull/9216#issuecomment-686720722
> Should I make an update to the readme @ryannedolan?
> https://github.com/apache/kafka/blob/trunk/connect/mirror/README.md doesn't have a section for running MirrorSourceConnector in a general connect cluster, so should I add one?
That'd be greatly appreciated!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jthompson6 commented on pull request #9216: KAFKA-10428: Fix schema for header conversion in MirrorSourceTask.
Posted by GitBox <gi...@apache.org>.
jthompson6 commented on pull request #9216:
URL: https://github.com/apache/kafka/pull/9216#issuecomment-686703916
Should I make an update to the readme @ryannedolan?
https://github.com/apache/kafka/blob/trunk/connect/mirror/README.md doesn't have a section for running MirrorSourceConnector in a general connect cluster, so should I add one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jthompson6 closed pull request #9216: KAFKA-10428: Fix schema for header conversion in MirrorSourceTask.
Posted by GitBox <gi...@apache.org>.
jthompson6 closed pull request #9216:
URL: https://github.com/apache/kafka/pull/9216
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jthompson6 commented on pull request #9216: KAFKA-10428: Fix schema for header conversion in MirrorSourceTask.
Posted by GitBox <gi...@apache.org>.
jthompson6 commented on pull request #9216:
URL: https://github.com/apache/kafka/pull/9216#issuecomment-681056654
@ryannedolan are you the right person to tag here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jthompson6 commented on pull request #9216: KAFKA-10428: Fix schema for header conversion in MirrorSourceTask.
Posted by GitBox <gi...@apache.org>.
jthompson6 commented on pull request #9216:
URL: https://github.com/apache/kafka/pull/9216#issuecomment-686723258
Cool. Closing this, will make a new PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jthompson6 commented on pull request #9216: KAFKA-10428: Fix schema for header conversion in MirrorSourceTask.
Posted by GitBox <gi...@apache.org>.
jthompson6 commented on pull request #9216:
URL: https://github.com/apache/kafka/pull/9216#issuecomment-681165609
I see the header converter class is configurable in https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerConfig.java#L90, but it appears to be a whole cluster setting, when really we want to only set it for this connector.
Do you think it is a bug that SimpleHeaderConvertor#fromConnectHeader transforms a byte[] header, even is the schema is BYTES? Maybe that is a question for @rhauch ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org