You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Gyula Fóra <gy...@gmail.com> on 2020/03/05 08:31:16 UTC
How to override flink-conf parameters for SQL Client
Hi All!
I am trying to understand if there is any way to override flink
configuration parameters when starting the SQL Client.
It seems that the only way to pass any parameters is through the
environment yaml.
There I found 2 possible routes:
configuration: this doesn't work as it only sets Table specific configs
apparently, but maybe I am wrong.
deployment: I tried using dynamic properties options here but unfortunately
we normalize (lowercase) the YAML keys so it is impossible to pass options
like -yD or -D.
Does anyone have any suggestions?
Thanks
Gyula
Re: How to override flink-conf parameters for SQL Client
Posted by Gyula Fóra <gy...@gmail.com>.
I feel that the current configuration section of the environment file
assumes too much about what the user wants to configure.
Configuring Table specific options is one thing but there are certainly
many many cases where users are deploying jobs in a per-job-cluster mode and
being able to override certain Flink configs is very useful.
With the huge improvements to the StreamExecutionEnv configuration coming
in 1.10 this is even more so.
I would propose to introduce a differentiation logic between "hard-coded"
configs (the ones we have now) and arbitrary config parameters that should
override the Configuration coming from flink-conf.yaml.
it could look something like:
flink-config:
state.backend : ROCKSDB
......
table-config:
what we have now
On Fri, Mar 6, 2020 at 3:22 AM Jeff Zhang <zj...@gmail.com> wrote:
> There's 2 kinds of configuration: job level & cluster level. I am afraid
> we don't have document to differentiate that, it depends on how user
> understand these configuration. We may need to improve document on that.
>
> Kurt Young <yk...@gmail.com> 于2020年3月6日周五 上午8:34写道：
>
>> If you already have a running flink cluster and you want submit another
>> job to this cluster, then all the configurations
>> relates to process parameters like TM memory, slot number etc are not be
>> able to modify.
>>
>> Best,
>> Kurt
>>
>>
>> On Thu, Mar 5, 2020 at 11:08 PM Gyula Fóra <gy...@gmail.com> wrote:
>>
>>> Kurt can you please explain which conf parameters do you mean?
>>>
>>> In regular executions (Yarn for instance) we have dynamic config
>>> parameters overriding any flink-conf argument.
>>> So it is not about setting them in the user code but it should happen
>>> before the ClusterDescriptors are created (ie in the together with the
>>> CustomCommandLine logic)
>>>
>>> Gyula
>>>
>>> On Thu, Mar 5, 2020 at 3:49 PM Kurt Young <yk...@gmail.com> wrote:
>>>
>>>> IIRC the tricky thing here is not all the config options belong to
>>>> flink-conf.yaml can be adjust dynamically in user's program.
>>>> So it will end up like some of the configurations can be overridden but
>>>> some are not. The experience is not quite good for users.
>>>>
>>>> Best,
>>>> Kurt
>>>>
>>>>
>>>> On Thu, Mar 5, 2020 at 10:15 PM Jeff Zhang <zj...@gmail.com> wrote:
>>>>
>>>>> Hi Gyula,
>>>>>
>>>>> I am doing integration Flink with Zeppelin. One feature in Zeppelin is
>>>>> that user could override any features in flink-conf.yaml. (Actually any
>>>>> features here
>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html).
>>>>> Of course you can run flink sql in Zeppelin, and could also leverage other
>>>>> features of Zeppelin, like visualization.
>>>>>
>>>>> If you are interested, you could try the master branch of Zeppelin +
>>>>> this improvement PR
>>>>>
>>>>> https://github.com/apache/zeppelin
>>>>> https://github.com/apache/zeppelin/pull/3676
>>>>>
>>>>> https://github.com/apache/zeppelin/blob/master/docs/interpreter/flink.md
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午6:51写道：
>>>>>
>>>>>> I could basically list a few things I want to set (execution.target
>>>>>> for example), but it's fair to assume that I would like to be able to set
>>>>>> anything :)
>>>>>>
>>>>>> Gyula
>>>>>>
>>>>>> On Thu, Mar 5, 2020 at 11:35 AM Jingsong Li <ji...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Gyula,
>>>>>>>
>>>>>>> Maybe Blink planner has invoked
>>>>>>> "StreamExecutionEnvironment.configure", which planner do you use?
>>>>>>>
>>>>>>> But "StreamExecutionEnvironment.configure" is only for partial
>>>>>>> configuration, can not for all configuration in flink-conf.yaml.
>>>>>>> So what's the config do you want to set? I know some config like
>>>>>>> "taskmanager.network.blocking-shuffle.compression.enabled" can not set....
>>>>>>>
>>>>>>> Best,
>>>>>>> Jingsong Lee
>>>>>>>
>>>>>>> On Thu, Mar 5, 2020 at 6:17 PM Jark Wu <im...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Gyula,
>>>>>>>>
>>>>>>>> Flink configurations can be overrided via
>>>>>>>> `TableConfig#getConfiguration()`, however, SQL CLI only allows to set Table
>>>>>>>> specific configs.
>>>>>>>> I will think it as a bug/improvement of SQL CLI which should be
>>>>>>>> fixed in 1.10.1.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Jark
>>>>>>>>
>>>>>>>> On Thu, 5 Mar 2020 at 18:12, Gyula Fóra <gy...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Caizhi,
>>>>>>>>>
>>>>>>>>> This seems like a pretty big shortcoming for any
>>>>>>>>> multi-user/multi-app environment. I will open a jira for this.
>>>>>>>>>
>>>>>>>>> Gyula
>>>>>>>>>
>>>>>>>>> On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <ts...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Gyula.
>>>>>>>>>>
>>>>>>>>>> I'm afraid there is no way to override all Flink configurations
>>>>>>>>>> currently. SQL client yaml file can only override some of the Flink
>>>>>>>>>> configurations.
>>>>>>>>>>
>>>>>>>>>> Configuration entries indeed can only set Table specific configs,
>>>>>>>>>> while deployment entires are used to set the result fetching address and
>>>>>>>>>> port. There is currently no way to change the execution target from the SQL
>>>>>>>>>> client.
>>>>>>>>>>
>>>>>>>>>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午4:31写道：
>>>>>>>>>>
>>>>>>>>>>> Hi All!
>>>>>>>>>>>
>>>>>>>>>>> I am trying to understand if there is any way to override flink
>>>>>>>>>>> configuration parameters when starting the SQL Client.
>>>>>>>>>>>
>>>>>>>>>>> It seems that the only way to pass any parameters is through the
>>>>>>>>>>> environment yaml.
>>>>>>>>>>>
>>>>>>>>>>> There I found 2 possible routes:
>>>>>>>>>>>
>>>>>>>>>>> configuration: this doesn't work as it only sets Table specific
>>>>>>>>>>> configs apparently, but maybe I am wrong.
>>>>>>>>>>>
>>>>>>>>>>> deployment: I tried using dynamic properties options here but
>>>>>>>>>>> unfortunately we normalize (lowercase) the YAML keys so it is impossible to
>>>>>>>>>>> pass options like -yD or -D.
>>>>>>>>>>>
>>>>>>>>>>> Does anyone have any suggestions?
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Gyula
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best, Jingsong Lee
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Best Regards
>>>>>
>>>>> Jeff Zhang
>>>>>
>>>>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: How to override flink-conf parameters for SQL Client
Posted by Jeff Zhang <zj...@gmail.com>.
There's 2 kinds of configuration: job level & cluster level. I am afraid we
don't have document to differentiate that, it depends on how user
understand these configuration. We may need to improve document on that.
Kurt Young <yk...@gmail.com> 于2020年3月6日周五 上午8:34写道：
> If you already have a running flink cluster and you want submit another
> job to this cluster, then all the configurations
> relates to process parameters like TM memory, slot number etc are not be
> able to modify.
>
> Best,
> Kurt
>
>
> On Thu, Mar 5, 2020 at 11:08 PM Gyula Fóra <gy...@gmail.com> wrote:
>
>> Kurt can you please explain which conf parameters do you mean?
>>
>> In regular executions (Yarn for instance) we have dynamic config
>> parameters overriding any flink-conf argument.
>> So it is not about setting them in the user code but it should happen
>> before the ClusterDescriptors are created (ie in the together with the
>> CustomCommandLine logic)
>>
>> Gyula
>>
>> On Thu, Mar 5, 2020 at 3:49 PM Kurt Young <yk...@gmail.com> wrote:
>>
>>> IIRC the tricky thing here is not all the config options belong to
>>> flink-conf.yaml can be adjust dynamically in user's program.
>>> So it will end up like some of the configurations can be overridden but
>>> some are not. The experience is not quite good for users.
>>>
>>> Best,
>>> Kurt
>>>
>>>
>>> On Thu, Mar 5, 2020 at 10:15 PM Jeff Zhang <zj...@gmail.com> wrote:
>>>
>>>> Hi Gyula,
>>>>
>>>> I am doing integration Flink with Zeppelin. One feature in Zeppelin is
>>>> that user could override any features in flink-conf.yaml. (Actually any
>>>> features here
>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html).
>>>> Of course you can run flink sql in Zeppelin, and could also leverage other
>>>> features of Zeppelin, like visualization.
>>>>
>>>> If you are interested, you could try the master branch of Zeppelin +
>>>> this improvement PR
>>>>
>>>> https://github.com/apache/zeppelin
>>>> https://github.com/apache/zeppelin/pull/3676
>>>> https://github.com/apache/zeppelin/blob/master/docs/interpreter/flink.md
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午6:51写道：
>>>>
>>>>> I could basically list a few things I want to set (execution.target
>>>>> for example), but it's fair to assume that I would like to be able to set
>>>>> anything :)
>>>>>
>>>>> Gyula
>>>>>
>>>>> On Thu, Mar 5, 2020 at 11:35 AM Jingsong Li <ji...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Gyula,
>>>>>>
>>>>>> Maybe Blink planner has invoked
>>>>>> "StreamExecutionEnvironment.configure", which planner do you use?
>>>>>>
>>>>>> But "StreamExecutionEnvironment.configure" is only for partial
>>>>>> configuration, can not for all configuration in flink-conf.yaml.
>>>>>> So what's the config do you want to set? I know some config like
>>>>>> "taskmanager.network.blocking-shuffle.compression.enabled" can not set....
>>>>>>
>>>>>> Best,
>>>>>> Jingsong Lee
>>>>>>
>>>>>> On Thu, Mar 5, 2020 at 6:17 PM Jark Wu <im...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Gyula,
>>>>>>>
>>>>>>> Flink configurations can be overrided via
>>>>>>> `TableConfig#getConfiguration()`, however, SQL CLI only allows to set Table
>>>>>>> specific configs.
>>>>>>> I will think it as a bug/improvement of SQL CLI which should be
>>>>>>> fixed in 1.10.1.
>>>>>>>
>>>>>>> Best,
>>>>>>> Jark
>>>>>>>
>>>>>>> On Thu, 5 Mar 2020 at 18:12, Gyula Fóra <gy...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Caizhi,
>>>>>>>>
>>>>>>>> This seems like a pretty big shortcoming for any
>>>>>>>> multi-user/multi-app environment. I will open a jira for this.
>>>>>>>>
>>>>>>>> Gyula
>>>>>>>>
>>>>>>>> On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <ts...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Gyula.
>>>>>>>>>
>>>>>>>>> I'm afraid there is no way to override all Flink configurations
>>>>>>>>> currently. SQL client yaml file can only override some of the Flink
>>>>>>>>> configurations.
>>>>>>>>>
>>>>>>>>> Configuration entries indeed can only set Table specific configs,
>>>>>>>>> while deployment entires are used to set the result fetching address and
>>>>>>>>> port. There is currently no way to change the execution target from the SQL
>>>>>>>>> client.
>>>>>>>>>
>>>>>>>>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午4:31写道：
>>>>>>>>>
>>>>>>>>>> Hi All!
>>>>>>>>>>
>>>>>>>>>> I am trying to understand if there is any way to override flink
>>>>>>>>>> configuration parameters when starting the SQL Client.
>>>>>>>>>>
>>>>>>>>>> It seems that the only way to pass any parameters is through the
>>>>>>>>>> environment yaml.
>>>>>>>>>>
>>>>>>>>>> There I found 2 possible routes:
>>>>>>>>>>
>>>>>>>>>> configuration: this doesn't work as it only sets Table specific
>>>>>>>>>> configs apparently, but maybe I am wrong.
>>>>>>>>>>
>>>>>>>>>> deployment: I tried using dynamic properties options here but
>>>>>>>>>> unfortunately we normalize (lowercase) the YAML keys so it is impossible to
>>>>>>>>>> pass options like -yD or -D.
>>>>>>>>>>
>>>>>>>>>> Does anyone have any suggestions?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Gyula
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best, Jingsong Lee
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Best Regards
>>>>
>>>> Jeff Zhang
>>>>
>>>
--
Best Regards
Jeff Zhang
Re: How to override flink-conf parameters for SQL Client
Posted by Kurt Young <yk...@gmail.com>.
If you already have a running flink cluster and you want submit another job
to this cluster, then all the configurations
relates to process parameters like TM memory, slot number etc are not be
able to modify.
Best,
Kurt
On Thu, Mar 5, 2020 at 11:08 PM Gyula Fóra <gy...@gmail.com> wrote:
> Kurt can you please explain which conf parameters do you mean?
>
> In regular executions (Yarn for instance) we have dynamic config
> parameters overriding any flink-conf argument.
> So it is not about setting them in the user code but it should happen
> before the ClusterDescriptors are created (ie in the together with the
> CustomCommandLine logic)
>
> Gyula
>
> On Thu, Mar 5, 2020 at 3:49 PM Kurt Young <yk...@gmail.com> wrote:
>
>> IIRC the tricky thing here is not all the config options belong to
>> flink-conf.yaml can be adjust dynamically in user's program.
>> So it will end up like some of the configurations can be overridden but
>> some are not. The experience is not quite good for users.
>>
>> Best,
>> Kurt
>>
>>
>> On Thu, Mar 5, 2020 at 10:15 PM Jeff Zhang <zj...@gmail.com> wrote:
>>
>>> Hi Gyula,
>>>
>>> I am doing integration Flink with Zeppelin. One feature in Zeppelin is
>>> that user could override any features in flink-conf.yaml. (Actually any
>>> features here
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html).
>>> Of course you can run flink sql in Zeppelin, and could also leverage other
>>> features of Zeppelin, like visualization.
>>>
>>> If you are interested, you could try the master branch of Zeppelin +
>>> this improvement PR
>>>
>>> https://github.com/apache/zeppelin
>>> https://github.com/apache/zeppelin/pull/3676
>>> https://github.com/apache/zeppelin/blob/master/docs/interpreter/flink.md
>>>
>>>
>>>
>>>
>>>
>>>
>>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午6:51写道：
>>>
>>>> I could basically list a few things I want to set (execution.target for
>>>> example), but it's fair to assume that I would like to be able to set
>>>> anything :)
>>>>
>>>> Gyula
>>>>
>>>> On Thu, Mar 5, 2020 at 11:35 AM Jingsong Li <ji...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Gyula,
>>>>>
>>>>> Maybe Blink planner has invoked
>>>>> "StreamExecutionEnvironment.configure", which planner do you use?
>>>>>
>>>>> But "StreamExecutionEnvironment.configure" is only for partial
>>>>> configuration, can not for all configuration in flink-conf.yaml.
>>>>> So what's the config do you want to set? I know some config like
>>>>> "taskmanager.network.blocking-shuffle.compression.enabled" can not set....
>>>>>
>>>>> Best,
>>>>> Jingsong Lee
>>>>>
>>>>> On Thu, Mar 5, 2020 at 6:17 PM Jark Wu <im...@gmail.com> wrote:
>>>>>
>>>>>> Hi Gyula,
>>>>>>
>>>>>> Flink configurations can be overrided via
>>>>>> `TableConfig#getConfiguration()`, however, SQL CLI only allows to set Table
>>>>>> specific configs.
>>>>>> I will think it as a bug/improvement of SQL CLI which should be fixed
>>>>>> in 1.10.1.
>>>>>>
>>>>>> Best,
>>>>>> Jark
>>>>>>
>>>>>> On Thu, 5 Mar 2020 at 18:12, Gyula Fóra <gy...@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Caizhi,
>>>>>>>
>>>>>>> This seems like a pretty big shortcoming for any
>>>>>>> multi-user/multi-app environment. I will open a jira for this.
>>>>>>>
>>>>>>> Gyula
>>>>>>>
>>>>>>> On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <ts...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Gyula.
>>>>>>>>
>>>>>>>> I'm afraid there is no way to override all Flink configurations
>>>>>>>> currently. SQL client yaml file can only override some of the Flink
>>>>>>>> configurations.
>>>>>>>>
>>>>>>>> Configuration entries indeed can only set Table specific configs,
>>>>>>>> while deployment entires are used to set the result fetching address and
>>>>>>>> port. There is currently no way to change the execution target from the SQL
>>>>>>>> client.
>>>>>>>>
>>>>>>>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午4:31写道：
>>>>>>>>
>>>>>>>>> Hi All!
>>>>>>>>>
>>>>>>>>> I am trying to understand if there is any way to override flink
>>>>>>>>> configuration parameters when starting the SQL Client.
>>>>>>>>>
>>>>>>>>> It seems that the only way to pass any parameters is through the
>>>>>>>>> environment yaml.
>>>>>>>>>
>>>>>>>>> There I found 2 possible routes:
>>>>>>>>>
>>>>>>>>> configuration: this doesn't work as it only sets Table specific
>>>>>>>>> configs apparently, but maybe I am wrong.
>>>>>>>>>
>>>>>>>>> deployment: I tried using dynamic properties options here but
>>>>>>>>> unfortunately we normalize (lowercase) the YAML keys so it is impossible to
>>>>>>>>> pass options like -yD or -D.
>>>>>>>>>
>>>>>>>>> Does anyone have any suggestions?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Gyula
>>>>>>>>>
>>>>>>>>
>>>>>
>>>>> --
>>>>> Best, Jingsong Lee
>>>>>
>>>>
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
Re: How to override flink-conf parameters for SQL Client
Posted by Gyula Fóra <gy...@gmail.com>.
Kurt can you please explain which conf parameters do you mean?
In regular executions (Yarn for instance) we have dynamic config
parameters overriding any flink-conf argument.
So it is not about setting them in the user code but it should happen
before the ClusterDescriptors are created (ie in the together with the
CustomCommandLine logic)
Gyula
On Thu, Mar 5, 2020 at 3:49 PM Kurt Young <yk...@gmail.com> wrote:
> IIRC the tricky thing here is not all the config options belong to
> flink-conf.yaml can be adjust dynamically in user's program.
> So it will end up like some of the configurations can be overridden but
> some are not. The experience is not quite good for users.
>
> Best,
> Kurt
>
>
> On Thu, Mar 5, 2020 at 10:15 PM Jeff Zhang <zj...@gmail.com> wrote:
>
>> Hi Gyula,
>>
>> I am doing integration Flink with Zeppelin. One feature in Zeppelin is
>> that user could override any features in flink-conf.yaml. (Actually any
>> features here
>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html).
>> Of course you can run flink sql in Zeppelin, and could also leverage other
>> features of Zeppelin, like visualization.
>>
>> If you are interested, you could try the master branch of Zeppelin + this
>> improvement PR
>>
>> https://github.com/apache/zeppelin
>> https://github.com/apache/zeppelin/pull/3676
>> https://github.com/apache/zeppelin/blob/master/docs/interpreter/flink.md
>>
>>
>>
>>
>>
>>
>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午6:51写道：
>>
>>> I could basically list a few things I want to set (execution.target for
>>> example), but it's fair to assume that I would like to be able to set
>>> anything :)
>>>
>>> Gyula
>>>
>>> On Thu, Mar 5, 2020 at 11:35 AM Jingsong Li <ji...@gmail.com>
>>> wrote:
>>>
>>>> Hi Gyula,
>>>>
>>>> Maybe Blink planner has invoked "StreamExecutionEnvironment.configure",
>>>> which planner do you use?
>>>>
>>>> But "StreamExecutionEnvironment.configure" is only for partial
>>>> configuration, can not for all configuration in flink-conf.yaml.
>>>> So what's the config do you want to set? I know some config like
>>>> "taskmanager.network.blocking-shuffle.compression.enabled" can not set....
>>>>
>>>> Best,
>>>> Jingsong Lee
>>>>
>>>> On Thu, Mar 5, 2020 at 6:17 PM Jark Wu <im...@gmail.com> wrote:
>>>>
>>>>> Hi Gyula,
>>>>>
>>>>> Flink configurations can be overrided via
>>>>> `TableConfig#getConfiguration()`, however, SQL CLI only allows to set Table
>>>>> specific configs.
>>>>> I will think it as a bug/improvement of SQL CLI which should be fixed
>>>>> in 1.10.1.
>>>>>
>>>>> Best,
>>>>> Jark
>>>>>
>>>>> On Thu, 5 Mar 2020 at 18:12, Gyula Fóra <gy...@gmail.com> wrote:
>>>>>
>>>>>> Thanks Caizhi,
>>>>>>
>>>>>> This seems like a pretty big shortcoming for any multi-user/multi-app
>>>>>> environment. I will open a jira for this.
>>>>>>
>>>>>> Gyula
>>>>>>
>>>>>> On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <ts...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Gyula.
>>>>>>>
>>>>>>> I'm afraid there is no way to override all Flink configurations
>>>>>>> currently. SQL client yaml file can only override some of the Flink
>>>>>>> configurations.
>>>>>>>
>>>>>>> Configuration entries indeed can only set Table specific configs,
>>>>>>> while deployment entires are used to set the result fetching address and
>>>>>>> port. There is currently no way to change the execution target from the SQL
>>>>>>> client.
>>>>>>>
>>>>>>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午4:31写道：
>>>>>>>
>>>>>>>> Hi All!
>>>>>>>>
>>>>>>>> I am trying to understand if there is any way to override flink
>>>>>>>> configuration parameters when starting the SQL Client.
>>>>>>>>
>>>>>>>> It seems that the only way to pass any parameters is through the
>>>>>>>> environment yaml.
>>>>>>>>
>>>>>>>> There I found 2 possible routes:
>>>>>>>>
>>>>>>>> configuration: this doesn't work as it only sets Table specific
>>>>>>>> configs apparently, but maybe I am wrong.
>>>>>>>>
>>>>>>>> deployment: I tried using dynamic properties options here but
>>>>>>>> unfortunately we normalize (lowercase) the YAML keys so it is impossible to
>>>>>>>> pass options like -yD or -D.
>>>>>>>>
>>>>>>>> Does anyone have any suggestions?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Gyula
>>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> Best, Jingsong Lee
>>>>
>>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
Re: How to override flink-conf parameters for SQL Client
Posted by Kurt Young <yk...@gmail.com>.
IIRC the tricky thing here is not all the config options belong to
flink-conf.yaml can be adjust dynamically in user's program.
So it will end up like some of the configurations can be overridden but
some are not. The experience is not quite good for users.
Best,
Kurt
On Thu, Mar 5, 2020 at 10:15 PM Jeff Zhang <zj...@gmail.com> wrote:
> Hi Gyula,
>
> I am doing integration Flink with Zeppelin. One feature in Zeppelin is
> that user could override any features in flink-conf.yaml. (Actually any
> features here
> https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html).
> Of course you can run flink sql in Zeppelin, and could also leverage other
> features of Zeppelin, like visualization.
>
> If you are interested, you could try the master branch of Zeppelin + this
> improvement PR
>
> https://github.com/apache/zeppelin
> https://github.com/apache/zeppelin/pull/3676
> https://github.com/apache/zeppelin/blob/master/docs/interpreter/flink.md
>
>
>
>
>
>
> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午6:51写道：
>
>> I could basically list a few things I want to set (execution.target for
>> example), but it's fair to assume that I would like to be able to set
>> anything :)
>>
>> Gyula
>>
>> On Thu, Mar 5, 2020 at 11:35 AM Jingsong Li <ji...@gmail.com>
>> wrote:
>>
>>> Hi Gyula,
>>>
>>> Maybe Blink planner has invoked "StreamExecutionEnvironment.configure",
>>> which planner do you use?
>>>
>>> But "StreamExecutionEnvironment.configure" is only for partial
>>> configuration, can not for all configuration in flink-conf.yaml.
>>> So what's the config do you want to set? I know some config like
>>> "taskmanager.network.blocking-shuffle.compression.enabled" can not set....
>>>
>>> Best,
>>> Jingsong Lee
>>>
>>> On Thu, Mar 5, 2020 at 6:17 PM Jark Wu <im...@gmail.com> wrote:
>>>
>>>> Hi Gyula,
>>>>
>>>> Flink configurations can be overrided via
>>>> `TableConfig#getConfiguration()`, however, SQL CLI only allows to set Table
>>>> specific configs.
>>>> I will think it as a bug/improvement of SQL CLI which should be fixed
>>>> in 1.10.1.
>>>>
>>>> Best,
>>>> Jark
>>>>
>>>> On Thu, 5 Mar 2020 at 18:12, Gyula Fóra <gy...@gmail.com> wrote:
>>>>
>>>>> Thanks Caizhi,
>>>>>
>>>>> This seems like a pretty big shortcoming for any multi-user/multi-app
>>>>> environment. I will open a jira for this.
>>>>>
>>>>> Gyula
>>>>>
>>>>> On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <ts...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Gyula.
>>>>>>
>>>>>> I'm afraid there is no way to override all Flink configurations
>>>>>> currently. SQL client yaml file can only override some of the Flink
>>>>>> configurations.
>>>>>>
>>>>>> Configuration entries indeed can only set Table specific configs,
>>>>>> while deployment entires are used to set the result fetching address and
>>>>>> port. There is currently no way to change the execution target from the SQL
>>>>>> client.
>>>>>>
>>>>>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午4:31写道：
>>>>>>
>>>>>>> Hi All!
>>>>>>>
>>>>>>> I am trying to understand if there is any way to override flink
>>>>>>> configuration parameters when starting the SQL Client.
>>>>>>>
>>>>>>> It seems that the only way to pass any parameters is through the
>>>>>>> environment yaml.
>>>>>>>
>>>>>>> There I found 2 possible routes:
>>>>>>>
>>>>>>> configuration: this doesn't work as it only sets Table specific
>>>>>>> configs apparently, but maybe I am wrong.
>>>>>>>
>>>>>>> deployment: I tried using dynamic properties options here but
>>>>>>> unfortunately we normalize (lowercase) the YAML keys so it is impossible to
>>>>>>> pass options like -yD or -D.
>>>>>>>
>>>>>>> Does anyone have any suggestions?
>>>>>>>
>>>>>>> Thanks
>>>>>>> Gyula
>>>>>>>
>>>>>>
>>>
>>> --
>>> Best, Jingsong Lee
>>>
>>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: How to override flink-conf parameters for SQL Client
Posted by Jeff Zhang <zj...@gmail.com>.
Hi Gyula,
I am doing integration Flink with Zeppelin. One feature in Zeppelin is that
user could override any features in flink-conf.yaml. (Actually any features
here
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html).
Of course you can run flink sql in Zeppelin, and could also leverage other
features of Zeppelin, like visualization.
If you are interested, you could try the master branch of Zeppelin + this
improvement PR
https://github.com/apache/zeppelin
https://github.com/apache/zeppelin/pull/3676
https://github.com/apache/zeppelin/blob/master/docs/interpreter/flink.md
Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午6:51写道：
> I could basically list a few things I want to set (execution.target for
> example), but it's fair to assume that I would like to be able to set
> anything :)
>
> Gyula
>
> On Thu, Mar 5, 2020 at 11:35 AM Jingsong Li <ji...@gmail.com>
> wrote:
>
>> Hi Gyula,
>>
>> Maybe Blink planner has invoked "StreamExecutionEnvironment.configure",
>> which planner do you use?
>>
>> But "StreamExecutionEnvironment.configure" is only for partial
>> configuration, can not for all configuration in flink-conf.yaml.
>> So what's the config do you want to set? I know some config like
>> "taskmanager.network.blocking-shuffle.compression.enabled" can not set....
>>
>> Best,
>> Jingsong Lee
>>
>> On Thu, Mar 5, 2020 at 6:17 PM Jark Wu <im...@gmail.com> wrote:
>>
>>> Hi Gyula,
>>>
>>> Flink configurations can be overrided via
>>> `TableConfig#getConfiguration()`, however, SQL CLI only allows to set Table
>>> specific configs.
>>> I will think it as a bug/improvement of SQL CLI which should be fixed in
>>> 1.10.1.
>>>
>>> Best,
>>> Jark
>>>
>>> On Thu, 5 Mar 2020 at 18:12, Gyula Fóra <gy...@gmail.com> wrote:
>>>
>>>> Thanks Caizhi,
>>>>
>>>> This seems like a pretty big shortcoming for any multi-user/multi-app
>>>> environment. I will open a jira for this.
>>>>
>>>> Gyula
>>>>
>>>> On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <ts...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Gyula.
>>>>>
>>>>> I'm afraid there is no way to override all Flink configurations
>>>>> currently. SQL client yaml file can only override some of the Flink
>>>>> configurations.
>>>>>
>>>>> Configuration entries indeed can only set Table specific configs,
>>>>> while deployment entires are used to set the result fetching address and
>>>>> port. There is currently no way to change the execution target from the SQL
>>>>> client.
>>>>>
>>>>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午4:31写道：
>>>>>
>>>>>> Hi All!
>>>>>>
>>>>>> I am trying to understand if there is any way to override flink
>>>>>> configuration parameters when starting the SQL Client.
>>>>>>
>>>>>> It seems that the only way to pass any parameters is through the
>>>>>> environment yaml.
>>>>>>
>>>>>> There I found 2 possible routes:
>>>>>>
>>>>>> configuration: this doesn't work as it only sets Table specific
>>>>>> configs apparently, but maybe I am wrong.
>>>>>>
>>>>>> deployment: I tried using dynamic properties options here but
>>>>>> unfortunately we normalize (lowercase) the YAML keys so it is impossible to
>>>>>> pass options like -yD or -D.
>>>>>>
>>>>>> Does anyone have any suggestions?
>>>>>>
>>>>>> Thanks
>>>>>> Gyula
>>>>>>
>>>>>
>>
>> --
>> Best, Jingsong Lee
>>
>
--
Best Regards
Jeff Zhang
Re: How to override flink-conf parameters for SQL Client
Posted by Gyula Fóra <gy...@gmail.com>.
I could basically list a few things I want to set (execution.target for
example), but it's fair to assume that I would like to be able to set
anything :)
Gyula
On Thu, Mar 5, 2020 at 11:35 AM Jingsong Li <ji...@gmail.com> wrote:
> Hi Gyula,
>
> Maybe Blink planner has invoked "StreamExecutionEnvironment.configure",
> which planner do you use?
>
> But "StreamExecutionEnvironment.configure" is only for partial
> configuration, can not for all configuration in flink-conf.yaml.
> So what's the config do you want to set? I know some config like
> "taskmanager.network.blocking-shuffle.compression.enabled" can not set....
>
> Best,
> Jingsong Lee
>
> On Thu, Mar 5, 2020 at 6:17 PM Jark Wu <im...@gmail.com> wrote:
>
>> Hi Gyula,
>>
>> Flink configurations can be overrided via
>> `TableConfig#getConfiguration()`, however, SQL CLI only allows to set Table
>> specific configs.
>> I will think it as a bug/improvement of SQL CLI which should be fixed in
>> 1.10.1.
>>
>> Best,
>> Jark
>>
>> On Thu, 5 Mar 2020 at 18:12, Gyula Fóra <gy...@gmail.com> wrote:
>>
>>> Thanks Caizhi,
>>>
>>> This seems like a pretty big shortcoming for any multi-user/multi-app
>>> environment. I will open a jira for this.
>>>
>>> Gyula
>>>
>>> On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <ts...@gmail.com>
>>> wrote:
>>>
>>>> Hi Gyula.
>>>>
>>>> I'm afraid there is no way to override all Flink configurations
>>>> currently. SQL client yaml file can only override some of the Flink
>>>> configurations.
>>>>
>>>> Configuration entries indeed can only set Table specific configs, while
>>>> deployment entires are used to set the result fetching address and port.
>>>> There is currently no way to change the execution target from the SQL
>>>> client.
>>>>
>>>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午4:31写道：
>>>>
>>>>> Hi All!
>>>>>
>>>>> I am trying to understand if there is any way to override flink
>>>>> configuration parameters when starting the SQL Client.
>>>>>
>>>>> It seems that the only way to pass any parameters is through the
>>>>> environment yaml.
>>>>>
>>>>> There I found 2 possible routes:
>>>>>
>>>>> configuration: this doesn't work as it only sets Table specific
>>>>> configs apparently, but maybe I am wrong.
>>>>>
>>>>> deployment: I tried using dynamic properties options here but
>>>>> unfortunately we normalize (lowercase) the YAML keys so it is impossible to
>>>>> pass options like -yD or -D.
>>>>>
>>>>> Does anyone have any suggestions?
>>>>>
>>>>> Thanks
>>>>> Gyula
>>>>>
>>>>
>
> --
> Best, Jingsong Lee
>
Re: How to override flink-conf parameters for SQL Client
Posted by Jingsong Li <ji...@gmail.com>.
Hi Gyula,
Maybe Blink planner has invoked "StreamExecutionEnvironment.configure",
which planner do you use?
But "StreamExecutionEnvironment.configure" is only for partial
configuration, can not for all configuration in flink-conf.yaml.
So what's the config do you want to set? I know some config like
"taskmanager.network.blocking-shuffle.compression.enabled" can not set....
Best,
Jingsong Lee
On Thu, Mar 5, 2020 at 6:17 PM Jark Wu <im...@gmail.com> wrote:
> Hi Gyula,
>
> Flink configurations can be overrided via
> `TableConfig#getConfiguration()`, however, SQL CLI only allows to set Table
> specific configs.
> I will think it as a bug/improvement of SQL CLI which should be fixed in
> 1.10.1.
>
> Best,
> Jark
>
> On Thu, 5 Mar 2020 at 18:12, Gyula Fóra <gy...@gmail.com> wrote:
>
>> Thanks Caizhi,
>>
>> This seems like a pretty big shortcoming for any multi-user/multi-app
>> environment. I will open a jira for this.
>>
>> Gyula
>>
>> On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <ts...@gmail.com> wrote:
>>
>>> Hi Gyula.
>>>
>>> I'm afraid there is no way to override all Flink configurations
>>> currently. SQL client yaml file can only override some of the Flink
>>> configurations.
>>>
>>> Configuration entries indeed can only set Table specific configs, while
>>> deployment entires are used to set the result fetching address and port.
>>> There is currently no way to change the execution target from the SQL
>>> client.
>>>
>>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午4:31写道：
>>>
>>>> Hi All!
>>>>
>>>> I am trying to understand if there is any way to override flink
>>>> configuration parameters when starting the SQL Client.
>>>>
>>>> It seems that the only way to pass any parameters is through the
>>>> environment yaml.
>>>>
>>>> There I found 2 possible routes:
>>>>
>>>> configuration: this doesn't work as it only sets Table specific configs
>>>> apparently, but maybe I am wrong.
>>>>
>>>> deployment: I tried using dynamic properties options here but
>>>> unfortunately we normalize (lowercase) the YAML keys so it is impossible to
>>>> pass options like -yD or -D.
>>>>
>>>> Does anyone have any suggestions?
>>>>
>>>> Thanks
>>>> Gyula
>>>>
>>>
--
Best, Jingsong Lee
Re: How to override flink-conf parameters for SQL Client
Posted by Jark Wu <im...@gmail.com>.
Hi Gyula,
Flink configurations can be overrided via `TableConfig#getConfiguration()`,
however, SQL CLI only allows to set Table specific configs.
I will think it as a bug/improvement of SQL CLI which should be fixed in
1.10.1.
Best,
Jark
On Thu, 5 Mar 2020 at 18:12, Gyula Fóra <gy...@gmail.com> wrote:
> Thanks Caizhi,
>
> This seems like a pretty big shortcoming for any multi-user/multi-app
> environment. I will open a jira for this.
>
> Gyula
>
> On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <ts...@gmail.com> wrote:
>
>> Hi Gyula.
>>
>> I'm afraid there is no way to override all Flink configurations
>> currently. SQL client yaml file can only override some of the Flink
>> configurations.
>>
>> Configuration entries indeed can only set Table specific configs, while
>> deployment entires are used to set the result fetching address and port.
>> There is currently no way to change the execution target from the SQL
>> client.
>>
>> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午4:31写道：
>>
>>> Hi All!
>>>
>>> I am trying to understand if there is any way to override flink
>>> configuration parameters when starting the SQL Client.
>>>
>>> It seems that the only way to pass any parameters is through the
>>> environment yaml.
>>>
>>> There I found 2 possible routes:
>>>
>>> configuration: this doesn't work as it only sets Table specific configs
>>> apparently, but maybe I am wrong.
>>>
>>> deployment: I tried using dynamic properties options here but
>>> unfortunately we normalize (lowercase) the YAML keys so it is impossible to
>>> pass options like -yD or -D.
>>>
>>> Does anyone have any suggestions?
>>>
>>> Thanks
>>> Gyula
>>>
>>
Re: How to override flink-conf parameters for SQL Client
Posted by Gyula Fóra <gy...@gmail.com>.
Thanks Caizhi,
This seems like a pretty big shortcoming for any multi-user/multi-app
environment. I will open a jira for this.
Gyula
On Thu, Mar 5, 2020 at 10:58 AM Caizhi Weng <ts...@gmail.com> wrote:
> Hi Gyula.
>
> I'm afraid there is no way to override all Flink configurations currently.
> SQL client yaml file can only override some of the Flink configurations.
>
> Configuration entries indeed can only set Table specific configs, while
> deployment entires are used to set the result fetching address and port.
> There is currently no way to change the execution target from the SQL
> client.
>
> Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午4:31写道：
>
>> Hi All!
>>
>> I am trying to understand if there is any way to override flink
>> configuration parameters when starting the SQL Client.
>>
>> It seems that the only way to pass any parameters is through the
>> environment yaml.
>>
>> There I found 2 possible routes:
>>
>> configuration: this doesn't work as it only sets Table specific configs
>> apparently, but maybe I am wrong.
>>
>> deployment: I tried using dynamic properties options here but
>> unfortunately we normalize (lowercase) the YAML keys so it is impossible to
>> pass options like -yD or -D.
>>
>> Does anyone have any suggestions?
>>
>> Thanks
>> Gyula
>>
>
Re: How to override flink-conf parameters for SQL Client
Posted by Caizhi Weng <ts...@gmail.com>.
Hi Gyula.
I'm afraid there is no way to override all Flink configurations currently.
SQL client yaml file can only override some of the Flink configurations.
Configuration entries indeed can only set Table specific configs, while
deployment entires are used to set the result fetching address and port.
There is currently no way to change the execution target from the SQL
client.
Gyula Fóra <gy...@gmail.com> 于2020年3月5日周四 下午4:31写道：
> Hi All!
>
> I am trying to understand if there is any way to override flink
> configuration parameters when starting the SQL Client.
>
> It seems that the only way to pass any parameters is through the
> environment yaml.
>
> There I found 2 possible routes:
>
> configuration: this doesn't work as it only sets Table specific configs
> apparently, but maybe I am wrong.
>
> deployment: I tried using dynamic properties options here but
> unfortunately we normalize (lowercase) the YAML keys so it is impossible to
> pass options like -yD or -D.
>
> Does anyone have any suggestions?
>
> Thanks
> Gyula
>