You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jingsong Li <ji...@gmail.com> on 2020/05/19 02:46:07 UTC
Re: Is it possible to change 'connector.startup-mode' option in the
flink job
Hi Thomas,
Good to hear from you. This is a very common problem.
In 1.11, we have two FLIP to solve your problem. [1][2] You can take a look.
I think dynamic table options (table hints) is enough for your requirement.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE
Best,
Jingsong Lee
On Tue, May 19, 2020 at 10:37 AM Thomas Huang <ly...@hotmail.com> wrote:
> Hi guys,
>
> I'm using hive to store kafka topic metadata as follows::
>
> CREATE TABLE orders (
> user_id BIGINT,
> product STRING,
> order_time TIMESTAMP(3),
> WATERMARK FOR order_time AS order_time - '5' SECONDS) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = 'orders',
> 'connector.startup-mode' = 'earliest-offset',
> 'connector.properties.bootstrap.servers' = 'localhost:9092',
> 'format.type' = 'json' );
>
> However, sometimes for incident handle, I have to set
> 'connector.startup-mode' to a specific timestamp, but I don't want to
> change the option setting in the hive table. Is there any way (api or ddl)
> to change 'connector.startup-mode' option in the flink jobs, but not
> impact the option stores in hive?
>
>
> Best Wishes.
>
--
Best, Jingsong Lee
Re: Is it possible to change 'connector.startup-mode' option in the
flink job
Posted by Thomas Huang <ly...@hotmail.com>.
Hi Jingsong,
Cool, Thanks for your reply.
Best wishes.
________________________________
From: Jingsong Li <ji...@gmail.com>
Sent: Tuesday, May 19, 2020 10:46
To: Thomas Huang <ly...@hotmail.com>
Cc: Flink <us...@flink.apache.org>
Subject: Re: Is it possible to change 'connector.startup-mode' option in the flink job
Hi Thomas,
Good to hear from you. This is a very common problem.
In 1.11, we have two FLIP to solve your problem. [1][2] You can take a look.
I think dynamic table options (table hints) is enough for your requirement.
[1]https://cwiki.apache.org/confluence/display/FLINK/FLIP-113%3A+Supports+Dynamic+Table+Options+for+Flink+SQL
[2]https://cwiki.apache.org/confluence/display/FLINK/FLIP-110%3A+Support+LIKE+clause+in+CREATE+TABLE
Best,
Jingsong Lee
On Tue, May 19, 2020 at 10:37 AM Thomas Huang <ly...@hotmail.com>> wrote:
Hi guys,
I'm using hive to store kafka topic metadata as follows::
CREATE TABLE orders (
user_id BIGINT,
product STRING,
order_time TIMESTAMP(3),
WATERMARK FOR order_time AS order_time - '5' SECONDS
) WITH (
'connector.type' = 'kafka',
'connector.version' = 'universal',
'connector.topic' = 'orders',
'connector.startup-mode' = 'earliest-offset',
'connector.properties.bootstrap.servers' = 'localhost:9092',
'format.type' = 'json'
);
However, sometimes for incident handle, I have to set 'connector.startup-mode' to a specific timestamp, but I don't want to change the option setting in the hive table. Is there any way (api or ddl) to change 'connector.startup-mode' option in the flink jobs, but not impact the option stores in hive?
Best Wishes.
--
Best, Jingsong Lee