[jira] [Commented] (FLINK-12256) Implement Confluent Schema Registry Catalog

["Guangbin Zhu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-12256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091381#comment-17091381 ] 

Guangbin Zhu commented on FLINK-12256:
--------------------------------------

Hi, [~phoenixjiangnan],

    I'm interested in this JIRA. In my senario, we use Confluent Schema Registry heavily for restricting and versioning avro schema of kafka topic. Unfortunately, Flink now doest not support CSR avro format.

  At the very start, I use avro format , the job failed , exceptions like this:

 
{code:java}
 {code}
!Xnip2020-04-24_17-25-39.jpg|width=1015,height=320!

 

After research, I found that csr avro format is different than apache avro format. As you said, csr avro record is 5 bytes more ahead of origin avro record bytes (one byte magic, and 4 byte schemaId in csr). That is the AvroRowDeserializationSchema cannot deserialize csr avro record. I implemented a seperate csravro format , and it works correctly.

My commit is [here|[https://github.com/zhuguangbin/flink/commit/79192601094da90368b81cd59f57b4b31c87004e]] , based on flink-1.10.0 release version.

 

 

 

> Implement Confluent Schema Registry Catalog
> -------------------------------------------
>
>                 Key: FLINK-12256
>                 URL: https://issues.apache.org/jira/browse/FLINK-12256
>             Project: Flink
>          Issue Type: New Feature
>          Components: Connectors / Kafka, Table SQL / Client
>    Affects Versions: 1.9.0
>            Reporter: Artsem Semianenka
>            Assignee: Bowen Li
>            Priority: Major
>             Fix For: 1.11.0
>
>         Attachments: Xnip2020-04-24_17-25-39.jpg
>
>
>  KafkaReadableCatalog is a special implementation of ReadableCatalog interface (which introduced in [FLIP-30|https://cwiki.apache.org/confluence/display/FLINK/FLIP-30%3A+Unified+Catalog+APIs] )  to retrieve meta information such topic name/schema of the topic from Apache Kafka and Confluent Schema Registry. 
> New ReadableCatalog allows a user to run SQL queries like:
> {code:java}
> Select * form kafka.topic_name  
> {code}
> without the need for manual definition of the table schema.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)