You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Ammu Parvathy (Jira)" <ji...@apache.org> on 2024/08/23 10:37:00 UTC
[jira] [Commented] (FLINK-24494) Avro Confluent Registry SQL kafka connector fails to write to the topic with schema
[ https://issues.apache.org/jira/browse/FLINK-24494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876228#comment-17876228 ]
Ammu Parvathy commented on FLINK-24494:
---------------------------------------
[~mahen] I can see a similar issue FLINK-25962 and fix implemented through PR: [https://github.com/apache/flink/commit/2c58dca500d0ec4f5d80852aa96ddb9c06ae4d61#diff-cb50d028339399779895d52557ffdb9537a6f793f2ef30af141bcd7848186595R119].
Do you think this issue can be closed ?
> Avro Confluent Registry SQL kafka connector fails to write to the topic with schema
> -------------------------------------------------------------------------------------
>
> Key: FLINK-24494
> URL: https://issues.apache.org/jira/browse/FLINK-24494
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
> Affects Versions: 1.14.0, 1.13.2
> Reporter: Mahendran Ponnusamy
> Priority: Critical
> Attachments: Screen Shot 2021-10-12 at 10.38.55 AM.png, image-2021-10-12-12-09-30-374.png, image-2021-10-12-12-11-04-664.png, image-2021-10-12-12-18-53-016.png, image-2021-10-12-12-19-37-227.png, image-2021-10-12-12-21-02-008.png, image-2021-10-12-12-21-46-177.png
>
>
> *Summary:*
> Given a schema registered to a topic with name and namespace
> when the flink sql with upsert-kafka connector writes to the topic,
> it fails coz row it tries to produce is not compatible with the schema registered
>
> *Root cause:*
> The upsert-kafka connector auto generates a schema with the +*name as `record` and no namespace*+. The below schema is generated by the connector. I'm expecting the connector should pull the schema from the subject and use ConfluentAvroRowSerialization to[which is not there today i believe] serialize using the schema from the subject.
> Schema generated by the upsert-kafka connector which is using AvroRowSerializer interanally
> !image-2021-10-12-12-21-46-177.png|width=813,height=23!
> {color:#cc7832}Schema Registered to the subject: {color}
> {
> {color:#9876aa}"type" {color}{color:#cc7832}: {color}{color:#6a8759}"record"{color}{color:#cc7832},{color} {color:#9876aa}"name" {color}{color:#cc7832}: {color}{color:#6a8759}"SimpleCustomer"{color}{color:#cc7832},{color} {color:#9876aa}"namespace" {color}{color:#cc7832}: {color}{color:#6a8759}"com...example.model"{color}{color:#cc7832},{color} {color:#9876aa}"fields" {color}{color:#cc7832}: {color}[ {
> {color:#9876aa}"name" {color}{color:#cc7832}: {color}{color:#6a8759}"customerId"{color}{color:#cc7832},{color} {color:#9876aa}"type" {color}{color:#cc7832}: {color}{color:#6a8759}"string"{color} }{color:#cc7832}, {color}{
> {color:#9876aa}"name" {color}{color:#cc7832}: {color}{color:#6a8759}"age"{color}{color:#cc7832},{color} {color:#9876aa}"type" {color}{color:#cc7832}: {color}{color:#6a8759}"int"{color}{color:#cc7832},{color} {color:#9876aa}"default" {color}{color:#cc7832}: {color}{color:#6897bb}0{color} }]
> }
>
> Table SQL with upsert-kafka connector
> !image-2021-10-12-12-18-53-016.png|width=351,height=176!
>
> The name "record" hardcoded
> !image-2021-10-12-12-21-02-008.png|width=464,height=138!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)