[jira] [Created] (FLINK-23777) Support dynamic parameters for sink table in multiple connectors

["zoucao (Jira)" <ji...@apache.org>]

zoucao created FLINK-23777:
------------------------------

             Summary: Support dynamic parameters for sink table in multiple connectors
                 Key: FLINK-23777
                 URL: https://issues.apache.org/jira/browse/FLINK-23777
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / ElasticSearch, Connectors / Kafka
            Reporter: zoucao


For now, metadata has been supported in SQL, it is simple, elegant and easy to use. 

But it is mostly used to get meta info from source table for now, I think we could use it more to support dynamic parameters for sink table. 

For example,
 * In kafka sink, we can use metadata to achieve dynamic partitions，let record choose it's partition by calculation logic.
{code:java}
CREATE TABLE KafkaSinkTable (
  `user_id` BIGINT,
  `item_id` BIGINT,
  `behavior` STRING,
  `partition` INT METADATA FROM 'partition'
) WITH (
  'connector' = 'kafka'
)
{code}

 * In Elasticsearch sink,  we also can use metadata to support dynamic index, like
{code:java}
CREATE TABLE ESSinkTable (
  index  VARCHAR  METADATA FROM 'index',
  log_ts VARCHAR
) with (
  'connector' = 'elasticsearch-7'
)
{code}
which is more elegant than treating a specific field as partial index like `myusers-\{log_ts|yyyy-MM-dd}`.

If there are some opinions or  other dynamic parameters worth to do, it could be propsed and dicussed here.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)