[jira] [Created] (FLINK-15985) offload runtime params from DDL to table hints in DML/queries

["Bowen Li (Jira)" <ji...@apache.org>]

Bowen Li created FLINK-15985:
--------------------------------

             Summary: offload runtime params from DDL to table hints in DML/queries
                 Key: FLINK-15985
                 URL: https://issues.apache.org/jira/browse/FLINK-15985
             Project: Flink
          Issue Type: New Feature
          Components: Table SQL / API
            Reporter: Bowen Li
            Assignee: Danny Chen
             Fix For: 1.11.0


background:

Currently Flink DDL mixes three types of params all together: 
 * External data’s metadata: defines what the data looks like (schema), where it is (location/url), how it should be accessed (username/pwd)
 * Source/sink runtime params: defines how and usually how fast Flink source/sink reads/writes data, not affecting the results
 * Kafka “sink-partitioner”
 * Elastic “bulk-flush.interval/max-size/...”


 * Semantics params: defines aspects like how much data Flink reads/writes, how the result will look like
 * Kafka “startup-mode”, “offset”
 * Watermark, timestamp column

 

Problems of the current mix-up: Flink cannot leverage catalogs and external system metadata alone to run queries with all the non-metadata params involved in DDL. E.g. when we add a catalog for Confluent Schema Registry, the expected user experience should be that Flink users just configure the catalog with url and usr/pwd, and should be able to run queries immediately; however, that’s not the case right now because users still have to use DDL to define a bunch params like “startup-mode”, “offset”, timestamp column, etc, along with the schema redundantly. We’ve heard many user complaints on this.

 

cc [~ykt836] [~lirui] [~lzljs3620320] [~jark] [~twalthr] [~dwysakowicz]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)