You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Paul Lam <pa...@gmail.com> on 2021/09/23 16:24:31 UTC
Relation between Flink Configuration and TableEnv
Hi all,
Currently, Flink creates a new Configuration in TableConfig of
StreamTableEnvironment, and synchronizes options in it back to the
Configuration of the underlying StreamExecutionEnvironment afterward.
However, only "relevant" options are set back [1], others are dropped
silently. That blocks users from overriding options like
DeploymentOptions.ATTACHED from TableConfig. So I'm wondering the reason
why not merge all options from TableConfig into the Configuration, and
what's the relation between Configuration and TableEnv?
Thanks a lot!
[1]
https://github.com/apache/flink/blob/36ff71f5ff63a140acc634dd1d98b2bb47a76ba5/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L904
Best,
Paul Lam
Re: Relation between Flink Configuration and TableEnv
Posted by Paul Lam <pa...@gmail.com>.
Hi Caizhi,
Thanks a lot for you clarification! Now I understand the design of TableConfig.
Best,
Paul Lam
> 2021年9月24日 15:40，Caizhi Weng <ts...@gmail.com> 写道：
>
> Hi!
>
> TableConfig is for configurations related to the Table and SQL API, especially the configurations in OptimizerConfigOptions and ExecutionConfigOptions.
>
> By Flink Configuration I guess you mean the configuration in Configuration. Sadly, as you say, it can be configured only once when creating the execution environment, and only a few configuration can be modified with the StreamExecutionEnvironment#configure method. This configuration contains datastream and execution related configurations.
>
> I bet the devs separate the configurations because they belong to different modules, but yes sometimes users will set their configuration in the wrong place.
>
> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> 于2021年9月24日周五 上午12:26写道：
> Sorry, I mean the relation between Flink Configuration and TableConfig, not TableEnv.
>
> Best,
> Paul Lam
>
> Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> 于2021年9月24日周五 上午12:24写道：
> Hi all,
>
> Currently, Flink creates a new Configuration in TableConfig of StreamTableEnvironment, and synchronizes options in it back to the Configuration of the underlying StreamExecutionEnvironment afterward. However, only "relevant" options are set back [1], others are dropped silently. That blocks users from overriding options like DeploymentOptions.ATTACHED from TableConfig. So I'm wondering the reason why not merge all options from TableConfig into the Configuration, and what's the relation between Configuration and TableEnv?
>
> Thanks a lot!
>
> [1] https://github.com/apache/flink/blob/36ff71f5ff63a140acc634dd1d98b2bb47a76ba5/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L904 <https://github.com/apache/flink/blob/36ff71f5ff63a140acc634dd1d98b2bb47a76ba5/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L904>
>
> Best,
> Paul Lam
>
Re: Relation between Flink Configuration and TableEnv
Posted by Caizhi Weng <ts...@gmail.com>.
Hi!
TableConfig is for configurations related to the Table and SQL API,
especially the configurations in OptimizerConfigOptions and
ExecutionConfigOptions.
By Flink Configuration I guess you mean the configuration in Configuration.
Sadly, as you say, it can be configured only once when creating the
execution environment, and only a few configuration can be modified with
the StreamExecutionEnvironment#configure method. This configuration
contains datastream and execution related configurations.
I bet the devs separate the configurations because they belong to different
modules, but yes sometimes users will set their configuration in the wrong
place.
Paul Lam <pa...@gmail.com> 于2021年9月24日周五 上午12:26写道：
> Sorry, I mean the relation between Flink Configuration and TableConfig,
> not TableEnv.
>
> Best,
> Paul Lam
>
> Paul Lam <pa...@gmail.com> 于2021年9月24日周五 上午12:24写道：
>
>> Hi all,
>>
>> Currently, Flink creates a new Configuration in TableConfig of
>> StreamTableEnvironment, and synchronizes options in it back to the
>> Configuration of the underlying StreamExecutionEnvironment afterward.
>> However, only "relevant" options are set back [1], others are dropped
>> silently. That blocks users from overriding options like
>> DeploymentOptions.ATTACHED from TableConfig. So I'm wondering the reason
>> why not merge all options from TableConfig into the Configuration, and
>> what's the relation between Configuration and TableEnv?
>>
>> Thanks a lot!
>>
>> [1]
>> https://github.com/apache/flink/blob/36ff71f5ff63a140acc634dd1d98b2bb47a76ba5/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L904
>>
>> Best,
>> Paul Lam
>>
>>
Re: Relation between Flink Configuration and TableEnv
Posted by Paul Lam <pa...@gmail.com>.
Sorry, I mean the relation between Flink Configuration and TableConfig, not
TableEnv.
Best,
Paul Lam
Paul Lam <pa...@gmail.com> 于2021年9月24日周五 上午12:24写道：
> Hi all,
>
> Currently, Flink creates a new Configuration in TableConfig of
> StreamTableEnvironment, and synchronizes options in it back to the
> Configuration of the underlying StreamExecutionEnvironment afterward.
> However, only "relevant" options are set back [1], others are dropped
> silently. That blocks users from overriding options like
> DeploymentOptions.ATTACHED from TableConfig. So I'm wondering the reason
> why not merge all options from TableConfig into the Configuration, and
> what's the relation between Configuration and TableEnv?
>
> Thanks a lot!
>
> [1]
> https://github.com/apache/flink/blob/36ff71f5ff63a140acc634dd1d98b2bb47a76ba5/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/environment/StreamExecutionEnvironment.java#L904
>
> Best,
> Paul Lam
>
>