[jira] [Commented] (FLINK-19517) Support for Confluent Kafka of Table Creation in Flink SQL Client

["Jark Wu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-19517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209381#comment-17209381 ] 

Jark Wu commented on FLINK-19517:
---------------------------------

Hi [~ksp0422], what's the difference between "Kafka" and "Confluent Kafka"? If the only difference is the confluent avro format, that has been supported in the 1.12 which allows you to define "avro-confluent" format and "avro-confluent.schema-registry.url", you can see the documentation for more details: https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/formats/avro-confluent.html

Regarding to 
1) "parallelism" configuration
This is under discussion, please watch the discuss thread in dev mailing list: 

2) specify custom properties within WITH
This has been supported in "kafka" connector by the "properties.*" options. 

3) have remote access to SQL client in cluster from local environment
I guess this is SQL gateway? Could you check whether this project [flink-sql-gateway|https://github.com/ververica/flink-sql-gateway] is what you are looking for?

> Support for Confluent Kafka of Table Creation in Flink SQL Client
> -----------------------------------------------------------------
>
>                 Key: FLINK-19517
>                 URL: https://issues.apache.org/jira/browse/FLINK-19517
>             Project: Flink
>          Issue Type: Wish
>    Affects Versions: 1.12.0
>            Reporter: Kevin Kwon
>            Priority: Critical
>
> Currently, table creation from SQL client such as below works well
> {code:sql}
> CREATE TABLE kafkaTable (
>   user_id BIGINT,
>   item_id BIGINT,
>   category_id BIGINT,
>   behavior STRING,
>   ts TIMESTAMP(3)
> ) WITH (
>   'connector' = 'kafka',
>   'topic' = 'user_behavior',
>   'properties.bootstrap.servers' = 'localhost:9092',
>   'properties.group.id' = 'testGroup',
>   'format' = 'avro',
>   'scan.startup.mode' = 'earliest-offset'
> )
> {code}
> Although I would wish for the table creation to support Confluent Kafka configuration as well. For example something like
> {code:sql}
> CREATE TABLE kafkaTable (
>   user_id BIGINT,
>   item_id BIGINT,
>   category_id BIGINT,
>   behavior STRING,
>   ts TIMESTAMP(3)
> ) WITH (
>   'connector' = 'confluent-kafka',
>   'topic' = 'user_behavior',
>   'properties.bootstrap.servers' = 'localhost:9092',
>   'properties.group.id' = 'testGroup',
>   'schema-registry' = 'http://schema-registry.com',
>   'scan.startup.mode' = 'earliest-offset'
> )
> {code}
> If this is enabled, it will be much more convenient to test queries on-the-fly that business analysts want to test against with 'Confluent Kafka'
> Additionally, it will be better if we can
>  - specify 'parallelism' within WITH clause to support parallel partition processing
>  - specify custom properties within WITH clause specified in [https://docs.confluent.io/5.4.2/installation/configuration/consumer-configs.html]
>  - have remote access to SQL client in cluster from local environment



--
This message was sent by Atlassian Jira
(v8.3.4#803005)