You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by zhengyu chen <ja...@gmail.com> on 2022/08/15 10:06:18 UTC
[DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Hi all,
We would like to start a discussion thread on FLIP-256 Support Job Dynamic
Parameter With Flink Rest Api
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api>
[1]
After the user submits the jar package, running a job through restapi
(/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
savepointPath, programArg, entry-class, parallelism) parameters, which is
obvious with the diversification of job parameters (eg Checkpoint address)
This solves the problem that the user can pass in other parameters when
submitting a job, avoiding the user to define these job parameters in the
code, resulting in the need to repackage the job for each modification
There was some interest from users [3] from a meetup and the mailing list.
Looking forward to comments and feedback, thanks!
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
[3] https://issues.apache.org/jira/browse/FLINK-27060
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by yanfei lei <fr...@gmail.com>.
+1, I think this is a very useful proposal, especially for session mode.
And for simplicity, maybe we can treat the job-level configurations and
cluster-level configurations identically, as your mentioned, the
cluster-level configurations will be ignored.
Best,
Yanfei
zhengyu chen <ja...@gmail.com> 于2022年8月15日周一 18:11写道：
> Hi all,
>
>
> We would like to start a discussion thread on FLIP-256 Support Job Dynamic
> Parameter With Flink Rest Api
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> >
> [1]
>
> After the user submits the jar package, running a job through restapi
> (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> savepointPath, programArg, entry-class, parallelism) parameters, which is
> obvious with the diversification of job parameters (eg Checkpoint address)
>
> This solves the problem that the user can pass in other parameters when
> submitting a job, avoiding the user to define these job parameters in the
> code, resulting in the need to repackage the job for each modification
>
> There was some interest from users [3] from a meetup and the mailing list.
> Looking forward to comments and feedback, thanks!
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
>
>
> [2]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
>
> [3] https://issues.apache.org/jira/browse/FLINK-27060
>
>
>
>
> --
> Best
>
> ConradJam
>
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Zheng Yu Chen <ja...@gmail.com>.
Everyone, we have 2 bounding votes and 2 unbounding votes in our voting
(url: https://lists.apache.org/thread/89f7kwnqgs8mfwos0h6b27lt3xn7383t).
According to the rules, we need at least 3 bound votes to start this FLIP.
If you forget to vote, it doesn't matter, we still have time to vote it,
let's do it
Peng Kang <ka...@shopee.com.invalid> 于2022年9月14日周三 10:05写道：
> +1
>
> On Sun, Sep 11, 2022 at 8:04 PM Zheng Yu Chen <ja...@gmail.com> wrote:
>
> > Now FLIP-256 is start vote in this thread url:
> >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread_89f7kwnqgs8mfwos0h6b27lt3xn7383t&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=SECfrK1ARoa03VLYzzznEE30H5w_zne5CCcjBFv0xIM&e=
> >
> >
> > zhengyu chen <ja...@gmail.com> 于2022年8月15日周一 18:06写道：
> >
> > >
> > > Hi all,
> > >
> > >
> > > We would like to start a discussion thread on FLIP-256 Support Job
> > > Dynamic Parameter With Flink Rest Api
> > > <
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_FLIP-2D256-2BSupport-2BJob-2BDynamic-2BParameter-2BWith-2BFlink-2BRest-2BApi&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=CBpiVI1TfwkD5hnDB-m6LlzobLn5_AFwUtru-_65xdA&e=
> > >
> > > [1]
> > >
> > > After the user submits the jar package, running a job through restapi
> > > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > > savepointPath, programArg, entry-class, parallelism) parameters, which
> is
> > > obvious with the diversification of job parameters (eg Checkpoint
> > > address)
> > >
> > > This solves the problem that the user can pass in other parameters when
> > > submitting a job, avoiding the user to define these job parameters in
> the
> > > code, resulting in the need to repackage the job for each modification
> > >
> > > There was some interest from users [3] from a meetup and the mailing
> > list.
> > > Looking forward to comments and feedback, thanks!
> > >
> > >
> > > [1]
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_FLIP-2D256-2BSupport-2BJob-2BDynamic-2BParameter-2BWith-2BFlink-2BRest-2BApi&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=CBpiVI1TfwkD5hnDB-m6LlzobLn5_AFwUtru-_65xdA&e=
> >
> > >
> > >
> > > [2]
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__nightlies.apache.org_flink_flink-2Ddocs-2Dmaster_docs_ops_rest-5Fapi_-23jars-2Djarid-2Drun&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=fgG0oof5F1cCG5a4wsrFhfBOpmdcso9qBGSxTjBwyxM&e=
> >
> > >
> > > [3]
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D27060&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=VAQa0d5RxLORKT_DhZkfRCf1M6YYbA_Yzl4Ut_vIyeE&e=
> >
> > >
> > >
> > >
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Peng Kang <ka...@shopee.com.INVALID>.
+1
On Sun, Sep 11, 2022 at 8:04 PM Zheng Yu Chen <ja...@gmail.com> wrote:
> Now FLIP-256 is start vote in this thread url:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread_89f7kwnqgs8mfwos0h6b27lt3xn7383t&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=SECfrK1ARoa03VLYzzznEE30H5w_zne5CCcjBFv0xIM&e=
>
>
> zhengyu chen <ja...@gmail.com> 于2022年8月15日周一 18:06写道：
>
> >
> > Hi all,
> >
> >
> > We would like to start a discussion thread on FLIP-256 Support Job
> > Dynamic Parameter With Flink Rest Api
> > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_FLIP-2D256-2BSupport-2BJob-2BDynamic-2BParameter-2BWith-2BFlink-2BRest-2BApi&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=CBpiVI1TfwkD5hnDB-m6LlzobLn5_AFwUtru-_65xdA&e=
> >
> > [1]
> >
> > After the user submits the jar package, running a job through restapi
> > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > savepointPath, programArg, entry-class, parallelism) parameters, which is
> > obvious with the diversification of job parameters (eg Checkpoint
> > address)
> >
> > This solves the problem that the user can pass in other parameters when
> > submitting a job, avoiding the user to define these job parameters in the
> > code, resulting in the need to repackage the job for each modification
> >
> > There was some interest from users [3] from a meetup and the mailing
> list.
> > Looking forward to comments and feedback, thanks!
> >
> >
> > [1]
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_FLIP-2D256-2BSupport-2BJob-2BDynamic-2BParameter-2BWith-2BFlink-2BRest-2BApi&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=CBpiVI1TfwkD5hnDB-m6LlzobLn5_AFwUtru-_65xdA&e=
>
> >
> >
> > [2]
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__nightlies.apache.org_flink_flink-2Ddocs-2Dmaster_docs_ops_rest-5Fapi_-23jars-2Djarid-2Drun&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=fgG0oof5F1cCG5a4wsrFhfBOpmdcso9qBGSxTjBwyxM&e=
>
> >
> > [3]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D27060&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=VAQa0d5RxLORKT_DhZkfRCf1M6YYbA_Yzl4Ut_vIyeE&e=
>
> >
> >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>
>
> --
> Best
>
> ConradJam
>
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Zheng Yu Chen <ja...@gmail.com>.
Now FLIP-256 is start vote in this thread url:
https://lists.apache.org/thread/89f7kwnqgs8mfwos0h6b27lt3xn7383t
zhengyu chen <ja...@gmail.com> 于2022年8月15日周一 18:06写道：
>
> Hi all,
>
>
> We would like to start a discussion thread on FLIP-256 Support Job
> Dynamic Parameter With Flink Rest Api
> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api>
> [1]
>
> After the user submits the jar package, running a job through restapi
> (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> savepointPath, programArg, entry-class, parallelism) parameters, which is
> obvious with the diversification of job parameters (eg Checkpoint
> address)
>
> This solves the problem that the user can pass in other parameters when
> submitting a job, avoiding the user to define these job parameters in the
> code, resulting in the need to repackage the job for each modification
>
> There was some interest from users [3] from a meetup and the mailing list.
> Looking forward to comments and feedback, thanks!
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
>
>
> [2]
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
>
> [3] https://issues.apache.org/jira/browse/FLINK-27060
>
>
>
>
> --
> Best
>
> ConradJam
>
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by zh...@outlook.com.
+1. Thanks for the effort. I think we should also extend the jarid-plan API [1] in a similar way, as some configurations might affect the jog graph as well.
[1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-plan
Best,
Zhanghao Chen
________________________________
From: zhengyu chen <ja...@gmail.com>
Sent: Monday, August 15, 2022 18:06
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Hi all,
We would like to start a discussion thread on FLIP-256 Support Job Dynamic
Parameter With Flink Rest Api
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api>
[1]
After the user submits the jar package, running a job through restapi
(/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
savepointPath, programArg, entry-class, parallelism) parameters, which is
obvious with the diversification of job parameters (eg Checkpoint address)
This solves the problem that the user can pass in other parameters when
submitting a job, avoiding the user to define these job parameters in the
code, resulting in the need to repackage the job for each modification
There was some interest from users [3] from a meetup and the mailing list.
Looking forward to comments and feedback, thanks!
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
[3] https://issues.apache.org/jira/browse/FLINK-27060
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Zheng Yu Chen <ja...@gmail.com>.
Hi Hong,
If we want to distinguish between "cluster configuration" and "job
configuration", I think the document will become complicated. What if
we can ensure that this part of the configuration does not take effect
or is intercepted?
When the user deploys Session Cluster, if the checkpoint interval (for
example, 10s) is configured in the flinkconfig file, and enabled (
jobmanager.rest.api.submit.job.allow-reset-config ),
Then when this parameter is passed in the jar/run api, it will be
blocked and prompted
how do you feel?
Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道：
>
> Hi Zheng Yu,
>
> We would have to take the same "cluster configuration" (cannot be set on job submission) vs "job configuration" approach in the User code as well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config as a cluster configuration. That way, neither the REST API / User code can override this configuration, and it can only be set in cluster configuration on startup.
>
> There are other cluster specific configurations that are already treated this way (e.g. jobmanager.rpc.address, jobmanager.memory.process.size). As part of this work, I wonder if we can update the Flink docs on configuration (https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/) to specify which configuration is "cluster" and which is "job".
>
> Regards,
> Hong
>
>
>
> On 20/08/2022, 09:16, "Zheng Yu Chen" <ja...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> @Gyula say that add a config for this but not expose it on the rest
> api is right
> when user start up session cluster,we can config (eg
> jobmanager.rest.api.submit.job.allow-reset-config = true/false) to
> controller this behavior
>
> Now we have an existing cluster, the admin configures checkpointing
> interval by this session cluster
>
> jobmanager.rest.api.submit.job.allow-reset-config = true/false
>
> - true : allow user reset new value config
> - false(default) : not allow user set new config from ,throw exp to
> tell user what properties is admin was set it
>
> However, there will be a problem in doing this. If the user writes
> this Flink Config in jar by hardcoding, the highest priority must be
> the code at this time, so the problem may still exist, and the user
> cannot be prevented from this behavior.
>
> what do you think ?
>
> @Hong @Gyula @Teoh @Danny
>
>
> Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
> >
> > +1 for the proposal.
> >
> > @Hong : I feel that the inverted override flag should be a cluster setting
> > and not something the user can override at will. I fear that this might
> > defeat the purpose of the feature itself.
> > So I think we should add a config for this but not expose it on the rest api
> >
> > Gyula
> >
> > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong <li...@amazon.co.uk.invalid>
> > wrote:
> >
> > > +1 to this FLIP.
> > > This is very useful for teams building a Flink platform to run jobs from
> > > an external user
> > >
> > > +1 on Danny's comment on adding a configuration to allow inverted order of
> > > overrides.
> > > However, might it be better to include an "override" toggle in the REST
> > > API itself? That way we can change the flink configuration override
> > > behaviour without restarting the Flink cluster. This would make sense if we
> > > are thinking of a Session cluster and deploying multiple Flink jobs to the
> > > same cluster.
> > >
> > > Regards,
> > > Hong
> > >
> > >
> > > On 18/08/2022, 10:46, "Danny Cranmer" <da...@apache.org> wrote:
> > >
> > > CAUTION: This email originated from outside of the organization. Do
> > > not click links or open attachments unless you can confirm the sender and
> > > know the content is safe.
> > >
> > >
> > >
> > > +1 thanks for driving this FLIP.
> > >
> > > We actually have an internally forked equivalent of this, which is on
> > > our
> > > list to try to upstream. Your proposal would *almost* work "off the
> > > shelf"
> > > for us. We have an inverted order or priority:
> > > - Rest API / Flink CLI > User Code > Cluster Config
> > >
> > > This is because our end users do not submit their jobs directly, our
> > > service does it on their behalf. For our usecase we do not want to
> > > allow
> > > users to override certain values we set, since some are managed from
> > > our
> > > service configuration, example: checkpointing interval.
> > >
> > > How would you feel about including a cluster configuration to invert
> > > the
> > > order? This could be decoupled to a follow-up FLIP.
> > >
> > > Thanks,
> > > Danny Cranmer
> > >
> > >
> > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <ja...@gmail.com>
> > > wrote:
> > >
> > > > This is a good suggestion, we can start this work after we finish the
> > > > discussion and vote
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > > >
> > > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > > >
> > > > > I've created a new ticket [FLINK-28973] Extending
> > > /jars/:jarid/plan API
> > > > to
> > > > > support setting Flink configs - ASF JIRA (apache.org)<
> > > > > https://issues.apache.org/jira/browse/FLINK-28973>. for the job
> > > plan
> > > > API.
> > > > > Maybe we can create a new umbrella ticket for FLIP-256 and put
> > > > FLINK-27060
> > > > > & FLINK-28973 as subtasks. WDYT?
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > ________________________________
> > > > > From: zhengyu chen <ja...@gmail.com>
> > > > > Sent: Monday, August 15, 2022 18:06
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With
> > > Flink Rest
> > > > > Api
> > > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > We would like to start a discussion thread on FLIP-256 Support Job
> > > > Dynamic
> > > > > Parameter With Flink Rest Api
> > > > > <
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > >
> > > > > [1]
> > > > >
> > > > > After the user submits the jar package, running a job through
> > > restapi
> > > > > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > > > > savepointPath, programArg, entry-class, parallelism) parameters,
> > > which is
> > > > > obvious with the diversification of job parameters (eg Checkpoint
> > > > address)
> > > > >
> > > > > This solves the problem that the user can pass in other parameters
> > > when
> > > > > submitting a job, avoiding the user to define these job parameters
> > > in the
> > > > > code, resulting in the need to repackage the job for each
> > > modification
> > > > >
> > > > > There was some interest from users [3] from a meetup and the
> > > mailing
> > > > list.
> > > > > Looking forward to comments and feedback, thanks!
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > >
> > > > >
> > > > > [2]
> > > > >
> > > > >
> > > >
> > > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > > >
> > > > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > >
> > >
> > >
>
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Gyula Fóra <gy...@gmail.com>.
Originally I assumed that user code programmatic config changes always have
highest priority in Flink. If I understand correctly this is true in other
deployment modes .
So I wouldn’t change that with a flag, the question is whether cluster
config or rest api config has a higher priority. I think the default could
be that rest api has a higher priority so:
(Default) override true:
User code > rest api > cluster conf
Override false :
User code > cluster conf > rest api
I might be wrong about the current behavior though…
Gyula
On Sun, 28 Aug 2022 at 07:45, Zheng Yu Chen <ja...@gmail.com> wrote:
> Hi Hong:
>
> Well, I think this should be the same idea as Gyula said before
> @Gyula What do you think ?
>
> If it is ok I will add this part to the documentation,Then the next process
> should be initiated by the community to vote this filp, I don't know who
> will host this part? Can someone help me with this 😀
>
>
> Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月27日周六 16:06写道：
>
> > Hi Zheng Yu,
> >
> > Sorry for the late reply, I was on holiday last week.
> >
> > Could we propose the following config instead?
> >
> > rest.submit.job.override-config = true/false
> > - true: REST API will have priority over user code (i.e. Rest API /
> > Flink CLI > User Code > Cluster Config)
> > - false (default): user code will have priority over REST API (i.e.
> User
> > Code > Rest API / Flink CLI > Cluster Config)
> >
> > This way, the default behaviour will be according to the proposed FLIP,
> > but we will have an additional toggle to ignore the configuration set in
> > user code.
> >
> > > However, there will be a problem in doing this. If the user
> writes
> > > this Flink Config in jar by hardcoding, the highest priority
> > must be
> > > the code at this time, so the problem may still exist, and the
> > user
> > > cannot be prevented from this behavior
> >
> > Hmm, would it be better if we set it such that the
> > "rest.submit.job.override-config" cannot be overwritten in user code
> > (ignored), just like configuration "jobmanager.rpc.port"?
> >
> > What do you think?
> >
> > Regards,
> > Hong
> >
> >
> >
> > On 26/08/2022, 12:05, "Zheng Yu Chen" <ja...@gmail.com> wrote:
> >
> > CAUTION: This email originated from outside of the organization. Do
> > not click links or open attachments unless you can confirm the sender and
> > know the content is safe.
> >
> >
> >
> > HI Hong,
> >
> > Maybe you forgot. I don’t know if the current FLIP still needs to be
> > modified. If not, we will start with the current plan. If you have
> > relevant ideas in the future, please continue to discuss. If not, we
> > will open voting at a later date.
> >
> > Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道：
> > >
> > > Hi Zheng Yu,
> > >
> > > We would have to take the same "cluster configuration" (cannot be
> > set on job submission) vs "job configuration" approach in the User code
> as
> > well. And we can classify
> jobmanager.rest.api.submit.job.allow-reset-config
> > as a cluster configuration. That way, neither the REST API / User code
> can
> > override this configuration, and it can only be set in cluster
> > configuration on startup.
> > >
> > > There are other cluster specific configurations that are already
> > treated this way (e.g. jobmanager.rpc.address,
> > jobmanager.memory.process.size). As part of this work, I wonder if we can
> > update the Flink docs on configuration (
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/
> )
> > to specify which configuration is "cluster" and which is "job".
> > >
> > > Regards,
> > > Hong
> > >
> > >
> > >
> > > On 20/08/2022, 09:16, "Zheng Yu Chen" <ja...@gmail.com> wrote:
> > >
> > > CAUTION: This email originated from outside of the
> organization.
> > Do not click links or open attachments unless you can confirm the sender
> > and know the content is safe.
> > >
> > >
> > >
> > > @Gyula say that add a config for this but not expose it on
> the
> > rest
> > > api is right
> > > when user start up session cluster,we can config (eg
> > > jobmanager.rest.api.submit.job.allow-reset-config = true/false)
> > to
> > > controller this behavior
> > >
> > > Now we have an existing cluster, the admin configures
> > checkpointing
> > > interval by this session cluster
> > >
> > > jobmanager.rest.api.submit.job.allow-reset-config = true/false
> > >
> > > - true : allow user reset new value config
> > > - false(default) : not allow user set new config from ,throw
> exp
> > to
> > > tell user what properties is admin was set it
> > >
> > > However, there will be a problem in doing this. If the user
> > writes
> > > this Flink Config in jar by hardcoding, the highest priority
> > must be
> > > the code at this time, so the problem may still exist, and the
> > user
> > > cannot be prevented from this behavior.
> > >
> > > what do you think ?
> > >
> > > @Hong @Gyula @Teoh @Danny
> > >
> > >
> > > Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
> > > >
> > > > +1 for the proposal.
> > > >
> > > > @Hong : I feel that the inverted override flag should be a
> > cluster setting
> > > > and not something the user can override at will. I fear that
> > this might
> > > > defeat the purpose of the feature itself.
> > > > So I think we should add a config for this but not expose it
> > on the rest api
> > > >
> > > > Gyula
> > > >
> > > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
> > <li...@amazon.co.uk.invalid>
> > > > wrote:
> > > >
> > > > > +1 to this FLIP.
> > > > > This is very useful for teams building a Flink platform to
> > run jobs from
> > > > > an external user
> > > > >
> > > > > +1 on Danny's comment on adding a configuration to allow
> > inverted order of
> > > > > overrides.
> > > > > However, might it be better to include an "override" toggle
> > in the REST
> > > > > API itself? That way we can change the flink configuration
> > override
> > > > > behaviour without restarting the Flink cluster. This would
> > make sense if we
> > > > > are thinking of a Session cluster and deploying multiple
> > Flink jobs to the
> > > > > same cluster.
> > > > >
> > > > > Regards,
> > > > > Hong
> > > > >
> > > > >
> > > > > On 18/08/2022, 10:46, "Danny Cranmer" <
> > dannycranmer@apache.org> wrote:
> > > > >
> > > > > CAUTION: This email originated from outside of the
> > organization. Do
> > > > > not click links or open attachments unless you can confirm
> > the sender and
> > > > > know the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > +1 thanks for driving this FLIP.
> > > > >
> > > > > We actually have an internally forked equivalent of
> > this, which is on
> > > > > our
> > > > > list to try to upstream. Your proposal would *almost*
> > work "off the
> > > > > shelf"
> > > > > for us. We have an inverted order or priority:
> > > > > - Rest API / Flink CLI > User Code > Cluster Config
> > > > >
> > > > > This is because our end users do not submit their jobs
> > directly, our
> > > > > service does it on their behalf. For our usecase we do
> > not want to
> > > > > allow
> > > > > users to override certain values we set, since some are
> > managed from
> > > > > our
> > > > > service configuration, example: checkpointing interval.
> > > > >
> > > > > How would you feel about including a cluster
> > configuration to invert
> > > > > the
> > > > > order? This could be decoupled to a follow-up FLIP.
> > > > >
> > > > > Thanks,
> > > > > Danny Cranmer
> > > > >
> > > > >
> > > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <
> > jam.gzczy@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This is a good suggestion, we can start this work
> > after we finish the
> > > > > > discussion and vote
> > > > > >
> > > > > > --
> > > > > > Best
> > > > > >
> > > > > > ConradJam
> > > > > >
> > > > > >
> > > > > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > > > > >
> > > > > > > I've created a new ticket [FLINK-28973] Extending
> > > > > /jars/:jarid/plan API
> > > > > > to
> > > > > > > support setting Flink configs - ASF JIRA (
> apache.org
> > )<
> > > > > > > https://issues.apache.org/jira/browse/FLINK-28973
> >.
> > for the job
> > > > > plan
> > > > > > API.
> > > > > > > Maybe we can create a new umbrella ticket for
> > FLIP-256 and put
> > > > > > FLINK-27060
> > > > > > > & FLINK-28973 as subtasks. WDYT?
> > > > > > >
> > > > > > > Best,
> > > > > > > Zhanghao Chen
> > > > > > > ________________________________
> > > > > > > From: zhengyu chen <ja...@gmail.com>
> > > > > > > Sent: Monday, August 15, 2022 18:06
> > > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic
> > Parameter With
> > > > > Flink Rest
> > > > > > > Api
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > >
> > > > > > > We would like to start a discussion thread on
> > FLIP-256 Support Job
> > > > > > Dynamic
> > > > > > > Parameter With Flink Rest Api
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > > After the user submits the jar package, running a
> > job through
> > > > > restapi
> > > > > > > (/jars/:jarid/run) [2] can only pass in
> > (allowNonRestoredState,
> > > > > > > savepointPath, programArg, entry-class,
> parallelism)
> > parameters,
> > > > > which is
> > > > > > > obvious with the diversification of job parameters
> > (eg Checkpoint
> > > > > > address)
> > > > > > >
> > > > > > > This solves the problem that the user can pass in
> > other parameters
> > > > > when
> > > > > > > submitting a job, avoiding the user to define these
> > job parameters
> > > > > in the
> > > > > > > code, resulting in the need to repackage the job
> for
> > each
> > > > > modification
> > > > > > >
> > > > > > > There was some interest from users [3] from a
> meetup
> > and the
> > > > > mailing
> > > > > > list.
> > > > > > > Looking forward to comments and feedback, thanks!
> > > > > > >
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > > >
> > > > > > >
> > > > > > > [2]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > > > > >
> > > > > > > [3]
> > https://issues.apache.org/jira/browse/FLINK-27060
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best
> > > > > > >
> > > > > > > ConradJam
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > >
> >
> > --
> > Best
> >
> > ConradJam
> >
> >
>
> --
> Best
>
> ConradJam
>
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Zheng Yu Chen <ja...@gmail.com>.
Hi Hong:
Well, I think this should be the same idea as Gyula said before
@Gyula What do you think ?
If it is ok I will add this part to the documentation,Then the next process
should be initiated by the community to vote this filp, I don't know who
will host this part? Can someone help me with this 😀
Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月27日周六 16:06写道：
> Hi Zheng Yu,
>
> Sorry for the late reply, I was on holiday last week.
>
> Could we propose the following config instead?
>
> rest.submit.job.override-config = true/false
> - true: REST API will have priority over user code (i.e. Rest API /
> Flink CLI > User Code > Cluster Config)
> - false (default): user code will have priority over REST API (i.e. User
> Code > Rest API / Flink CLI > Cluster Config)
>
> This way, the default behaviour will be according to the proposed FLIP,
> but we will have an additional toggle to ignore the configuration set in
> user code.
>
> > However, there will be a problem in doing this. If the user writes
> > this Flink Config in jar by hardcoding, the highest priority
> must be
> > the code at this time, so the problem may still exist, and the
> user
> > cannot be prevented from this behavior
>
> Hmm, would it be better if we set it such that the
> "rest.submit.job.override-config" cannot be overwritten in user code
> (ignored), just like configuration "jobmanager.rpc.port"?
>
> What do you think?
>
> Regards,
> Hong
>
>
>
> On 26/08/2022, 12:05, "Zheng Yu Chen" <ja...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> HI Hong,
>
> Maybe you forgot. I don’t know if the current FLIP still needs to be
> modified. If not, we will start with the current plan. If you have
> relevant ideas in the future, please continue to discuss. If not, we
> will open voting at a later date.
>
> Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道：
> >
> > Hi Zheng Yu,
> >
> > We would have to take the same "cluster configuration" (cannot be
> set on job submission) vs "job configuration" approach in the User code as
> well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config
> as a cluster configuration. That way, neither the REST API / User code can
> override this configuration, and it can only be set in cluster
> configuration on startup.
> >
> > There are other cluster specific configurations that are already
> treated this way (e.g. jobmanager.rpc.address,
> jobmanager.memory.process.size). As part of this work, I wonder if we can
> update the Flink docs on configuration (
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/)
> to specify which configuration is "cluster" and which is "job".
> >
> > Regards,
> > Hong
> >
> >
> >
> > On 20/08/2022, 09:16, "Zheng Yu Chen" <ja...@gmail.com> wrote:
> >
> > CAUTION: This email originated from outside of the organization.
> Do not click links or open attachments unless you can confirm the sender
> and know the content is safe.
> >
> >
> >
> > @Gyula say that add a config for this but not expose it on the
> rest
> > api is right
> > when user start up session cluster,we can config (eg
> > jobmanager.rest.api.submit.job.allow-reset-config = true/false)
> to
> > controller this behavior
> >
> > Now we have an existing cluster, the admin configures
> checkpointing
> > interval by this session cluster
> >
> > jobmanager.rest.api.submit.job.allow-reset-config = true/false
> >
> > - true : allow user reset new value config
> > - false(default) : not allow user set new config from ,throw exp
> to
> > tell user what properties is admin was set it
> >
> > However, there will be a problem in doing this. If the user
> writes
> > this Flink Config in jar by hardcoding, the highest priority
> must be
> > the code at this time, so the problem may still exist, and the
> user
> > cannot be prevented from this behavior.
> >
> > what do you think ?
> >
> > @Hong @Gyula @Teoh @Danny
> >
> >
> > Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
> > >
> > > +1 for the proposal.
> > >
> > > @Hong : I feel that the inverted override flag should be a
> cluster setting
> > > and not something the user can override at will. I fear that
> this might
> > > defeat the purpose of the feature itself.
> > > So I think we should add a config for this but not expose it
> on the rest api
> > >
> > > Gyula
> > >
> > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
> <li...@amazon.co.uk.invalid>
> > > wrote:
> > >
> > > > +1 to this FLIP.
> > > > This is very useful for teams building a Flink platform to
> run jobs from
> > > > an external user
> > > >
> > > > +1 on Danny's comment on adding a configuration to allow
> inverted order of
> > > > overrides.
> > > > However, might it be better to include an "override" toggle
> in the REST
> > > > API itself? That way we can change the flink configuration
> override
> > > > behaviour without restarting the Flink cluster. This would
> make sense if we
> > > > are thinking of a Session cluster and deploying multiple
> Flink jobs to the
> > > > same cluster.
> > > >
> > > > Regards,
> > > > Hong
> > > >
> > > >
> > > > On 18/08/2022, 10:46, "Danny Cranmer" <
> dannycranmer@apache.org> wrote:
> > > >
> > > > CAUTION: This email originated from outside of the
> organization. Do
> > > > not click links or open attachments unless you can confirm
> the sender and
> > > > know the content is safe.
> > > >
> > > >
> > > >
> > > > +1 thanks for driving this FLIP.
> > > >
> > > > We actually have an internally forked equivalent of
> this, which is on
> > > > our
> > > > list to try to upstream. Your proposal would *almost*
> work "off the
> > > > shelf"
> > > > for us. We have an inverted order or priority:
> > > > - Rest API / Flink CLI > User Code > Cluster Config
> > > >
> > > > This is because our end users do not submit their jobs
> directly, our
> > > > service does it on their behalf. For our usecase we do
> not want to
> > > > allow
> > > > users to override certain values we set, since some are
> managed from
> > > > our
> > > > service configuration, example: checkpointing interval.
> > > >
> > > > How would you feel about including a cluster
> configuration to invert
> > > > the
> > > > order? This could be decoupled to a follow-up FLIP.
> > > >
> > > > Thanks,
> > > > Danny Cranmer
> > > >
> > > >
> > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <
> jam.gzczy@gmail.com>
> > > > wrote:
> > > >
> > > > > This is a good suggestion, we can start this work
> after we finish the
> > > > > discussion and vote
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > > >
> > > > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > > > >
> > > > > > I've created a new ticket [FLINK-28973] Extending
> > > > /jars/:jarid/plan API
> > > > > to
> > > > > > support setting Flink configs - ASF JIRA (apache.org
> )<
> > > > > > https://issues.apache.org/jira/browse/FLINK-28973>.
> for the job
> > > > plan
> > > > > API.
> > > > > > Maybe we can create a new umbrella ticket for
> FLIP-256 and put
> > > > > FLINK-27060
> > > > > > & FLINK-28973 as subtasks. WDYT?
> > > > > >
> > > > > > Best,
> > > > > > Zhanghao Chen
> > > > > > ________________________________
> > > > > > From: zhengyu chen <ja...@gmail.com>
> > > > > > Sent: Monday, August 15, 2022 18:06
> > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic
> Parameter With
> > > > Flink Rest
> > > > > > Api
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > >
> > > > > > We would like to start a discussion thread on
> FLIP-256 Support Job
> > > > > Dynamic
> > > > > > Parameter With Flink Rest Api
> > > > > > <
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > > >
> > > > > > [1]
> > > > > >
> > > > > > After the user submits the jar package, running a
> job through
> > > > restapi
> > > > > > (/jars/:jarid/run) [2] can only pass in
> (allowNonRestoredState,
> > > > > > savepointPath, programArg, entry-class, parallelism)
> parameters,
> > > > which is
> > > > > > obvious with the diversification of job parameters
> (eg Checkpoint
> > > > > address)
> > > > > >
> > > > > > This solves the problem that the user can pass in
> other parameters
> > > > when
> > > > > > submitting a job, avoiding the user to define these
> job parameters
> > > > in the
> > > > > > code, resulting in the need to repackage the job for
> each
> > > > modification
> > > > > >
> > > > > > There was some interest from users [3] from a meetup
> and the
> > > > mailing
> > > > > list.
> > > > > > Looking forward to comments and feedback, thanks!
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > >
> > > > > >
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > > > >
> > > > > > [3]
> https://issues.apache.org/jira/browse/FLINK-27060
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best
> > > > > >
> > > > > > ConradJam
> > > > > >
> > > > >
> > > >
> > > >
> >
>
> --
> Best
>
> ConradJam
>
>
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by "Teoh, Hong" <li...@amazon.co.uk.INVALID>.
That sounds good to me. Thanks for working in this Zheng Yu!
Hong
________________________________________
From: Zheng Yu Chen <ja...@gmail.com>
Sent: 01 September 2022 05:08:03
To: dev@flink.apache.org
Subject: RE: [EXTERNAL][DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Hi @Hong
As @Chesnay said, we just follow the same order that the CLI currently
enforces. @
Gyula and me agree this sugget，
It also makes sense to mention that this FLIP is not about the
prioritization of config options. so this feature maybe do this filp and
after discuss , not now
After the addition is complete, if there are no new comments, we will enter
the voting stage and start it ~
what do you think?
Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月27日周六 16:06写道：
> Hi Zheng Yu,
>
> Sorry for the late reply, I was on holiday last week.
>
> Could we propose the following config instead?
>
> rest.submit.job.override-config = true/false
> - true: REST API will have priority over user code (i.e. Rest API /
> Flink CLI > User Code > Cluster Config)
> - false (default): user code will have priority over REST API (i.e. User
> Code > Rest API / Flink CLI > Cluster Config)
>
> This way, the default behaviour will be according to the proposed FLIP,
> but we will have an additional toggle to ignore the configuration set in
> user code.
>
> > However, there will be a problem in doing this. If the user writes
> > this Flink Config in jar by hardcoding, the highest priority
> must be
> > the code at this time, so the problem may still exist, and the
> user
> > cannot be prevented from this behavior
>
> Hmm, would it be better if we set it such that the
> "rest.submit.job.override-config" cannot be overwritten in user code
> (ignored), just like configuration "jobmanager.rpc.port"?
>
> What do you think?
>
> Regards,
> Hong
>
>
>
> On 26/08/2022, 12:05, "Zheng Yu Chen" <ja...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> HI Hong,
>
> Maybe you forgot. I don’t know if the current FLIP still needs to be
> modified. If not, we will start with the current plan. If you have
> relevant ideas in the future, please continue to discuss. If not, we
> will open voting at a later date.
>
> Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道：
> >
> > Hi Zheng Yu,
> >
> > We would have to take the same "cluster configuration" (cannot be
> set on job submission) vs "job configuration" approach in the User code as
> well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config
> as a cluster configuration. That way, neither the REST API / User code can
> override this configuration, and it can only be set in cluster
> configuration on startup.
> >
> > There are other cluster specific configurations that are already
> treated this way (e.g. jobmanager.rpc.address,
> jobmanager.memory.process.size). As part of this work, I wonder if we can
> update the Flink docs on configuration (
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/)
> to specify which configuration is "cluster" and which is "job".
> >
> > Regards,
> > Hong
> >
> >
> >
> > On 20/08/2022, 09:16, "Zheng Yu Chen" <ja...@gmail.com> wrote:
> >
> > CAUTION: This email originated from outside of the organization.
> Do not click links or open attachments unless you can confirm the sender
> and know the content is safe.
> >
> >
> >
> > @Gyula say that add a config for this but not expose it on the
> rest
> > api is right
> > when user start up session cluster,we can config (eg
> > jobmanager.rest.api.submit.job.allow-reset-config = true/false)
> to
> > controller this behavior
> >
> > Now we have an existing cluster, the admin configures
> checkpointing
> > interval by this session cluster
> >
> > jobmanager.rest.api.submit.job.allow-reset-config = true/false
> >
> > - true : allow user reset new value config
> > - false(default) : not allow user set new config from ,throw exp
> to
> > tell user what properties is admin was set it
> >
> > However, there will be a problem in doing this. If the user
> writes
> > this Flink Config in jar by hardcoding, the highest priority
> must be
> > the code at this time, so the problem may still exist, and the
> user
> > cannot be prevented from this behavior.
> >
> > what do you think ?
> >
> > @Hong @Gyula @Teoh @Danny
> >
> >
> > Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
> > >
> > > +1 for the proposal.
> > >
> > > @Hong : I feel that the inverted override flag should be a
> cluster setting
> > > and not something the user can override at will. I fear that
> this might
> > > defeat the purpose of the feature itself.
> > > So I think we should add a config for this but not expose it
> on the rest api
> > >
> > > Gyula
> > >
> > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
> <li...@amazon.co.uk.invalid>
> > > wrote:
> > >
> > > > +1 to this FLIP.
> > > > This is very useful for teams building a Flink platform to
> run jobs from
> > > > an external user
> > > >
> > > > +1 on Danny's comment on adding a configuration to allow
> inverted order of
> > > > overrides.
> > > > However, might it be better to include an "override" toggle
> in the REST
> > > > API itself? That way we can change the flink configuration
> override
> > > > behaviour without restarting the Flink cluster. This would
> make sense if we
> > > > are thinking of a Session cluster and deploying multiple
> Flink jobs to the
> > > > same cluster.
> > > >
> > > > Regards,
> > > > Hong
> > > >
> > > >
> > > > On 18/08/2022, 10:46, "Danny Cranmer" <
> dannycranmer@apache.org> wrote:
> > > >
> > > > CAUTION: This email originated from outside of the
> organization. Do
> > > > not click links or open attachments unless you can confirm
> the sender and
> > > > know the content is safe.
> > > >
> > > >
> > > >
> > > > +1 thanks for driving this FLIP.
> > > >
> > > > We actually have an internally forked equivalent of
> this, which is on
> > > > our
> > > > list to try to upstream. Your proposal would *almost*
> work "off the
> > > > shelf"
> > > > for us. We have an inverted order or priority:
> > > > - Rest API / Flink CLI > User Code > Cluster Config
> > > >
> > > > This is because our end users do not submit their jobs
> directly, our
> > > > service does it on their behalf. For our usecase we do
> not want to
> > > > allow
> > > > users to override certain values we set, since some are
> managed from
> > > > our
> > > > service configuration, example: checkpointing interval.
> > > >
> > > > How would you feel about including a cluster
> configuration to invert
> > > > the
> > > > order? This could be decoupled to a follow-up FLIP.
> > > >
> > > > Thanks,
> > > > Danny Cranmer
> > > >
> > > >
> > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <
> jam.gzczy@gmail.com>
> > > > wrote:
> > > >
> > > > > This is a good suggestion, we can start this work
> after we finish the
> > > > > discussion and vote
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > > >
> > > > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > > > >
> > > > > > I've created a new ticket [FLINK-28973] Extending
> > > > /jars/:jarid/plan API
> > > > > to
> > > > > > support setting Flink configs - ASF JIRA (apache.org
> )<
> > > > > > https://issues.apache.org/jira/browse/FLINK-28973>.
> for the job
> > > > plan
> > > > > API.
> > > > > > Maybe we can create a new umbrella ticket for
> FLIP-256 and put
> > > > > FLINK-27060
> > > > > > & FLINK-28973 as subtasks. WDYT?
> > > > > >
> > > > > > Best,
> > > > > > Zhanghao Chen
> > > > > > ________________________________
> > > > > > From: zhengyu chen <ja...@gmail.com>
> > > > > > Sent: Monday, August 15, 2022 18:06
> > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic
> Parameter With
> > > > Flink Rest
> > > > > > Api
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > >
> > > > > > We would like to start a discussion thread on
> FLIP-256 Support Job
> > > > > Dynamic
> > > > > > Parameter With Flink Rest Api
> > > > > > <
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > > >
> > > > > > [1]
> > > > > >
> > > > > > After the user submits the jar package, running a
> job through
> > > > restapi
> > > > > > (/jars/:jarid/run) [2] can only pass in
> (allowNonRestoredState,
> > > > > > savepointPath, programArg, entry-class, parallelism)
> parameters,
> > > > which is
> > > > > > obvious with the diversification of job parameters
> (eg Checkpoint
> > > > > address)
> > > > > >
> > > > > > This solves the problem that the user can pass in
> other parameters
> > > > when
> > > > > > submitting a job, avoiding the user to define these
> job parameters
> > > > in the
> > > > > > code, resulting in the need to repackage the job for
> each
> > > > modification
> > > > > >
> > > > > > There was some interest from users [3] from a meetup
> and the
> > > > mailing
> > > > > list.
> > > > > > Looking forward to comments and feedback, thanks!
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > >
> > > > > >
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > > > >
> > > > > > [3]
> https://issues.apache.org/jira/browse/FLINK-27060
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best
> > > > > >
> > > > > > ConradJam
> > > > > >
> > > > >
> > > >
> > > >
> >
>
> --
> Best
>
> ConradJam
>
>
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Zheng Yu Chen <ja...@gmail.com>.
Hi @Hong
As @Chesnay said, we just follow the same order that the CLI currently
enforces. @
Gyula and me agree this sugget，
It also makes sense to mention that this FLIP is not about the
prioritization of config options. so this feature maybe do this filp and
after discuss , not now
After the addition is complete, if there are no new comments, we will enter
the voting stage and start it ~
what do you think?
Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月27日周六 16:06写道：
> Hi Zheng Yu,
>
> Sorry for the late reply, I was on holiday last week.
>
> Could we propose the following config instead?
>
> rest.submit.job.override-config = true/false
> - true: REST API will have priority over user code (i.e. Rest API /
> Flink CLI > User Code > Cluster Config)
> - false (default): user code will have priority over REST API (i.e. User
> Code > Rest API / Flink CLI > Cluster Config)
>
> This way, the default behaviour will be according to the proposed FLIP,
> but we will have an additional toggle to ignore the configuration set in
> user code.
>
> > However, there will be a problem in doing this. If the user writes
> > this Flink Config in jar by hardcoding, the highest priority
> must be
> > the code at this time, so the problem may still exist, and the
> user
> > cannot be prevented from this behavior
>
> Hmm, would it be better if we set it such that the
> "rest.submit.job.override-config" cannot be overwritten in user code
> (ignored), just like configuration "jobmanager.rpc.port"?
>
> What do you think?
>
> Regards,
> Hong
>
>
>
> On 26/08/2022, 12:05, "Zheng Yu Chen" <ja...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> HI Hong,
>
> Maybe you forgot. I don’t know if the current FLIP still needs to be
> modified. If not, we will start with the current plan. If you have
> relevant ideas in the future, please continue to discuss. If not, we
> will open voting at a later date.
>
> Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道：
> >
> > Hi Zheng Yu,
> >
> > We would have to take the same "cluster configuration" (cannot be
> set on job submission) vs "job configuration" approach in the User code as
> well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config
> as a cluster configuration. That way, neither the REST API / User code can
> override this configuration, and it can only be set in cluster
> configuration on startup.
> >
> > There are other cluster specific configurations that are already
> treated this way (e.g. jobmanager.rpc.address,
> jobmanager.memory.process.size). As part of this work, I wonder if we can
> update the Flink docs on configuration (
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/)
> to specify which configuration is "cluster" and which is "job".
> >
> > Regards,
> > Hong
> >
> >
> >
> > On 20/08/2022, 09:16, "Zheng Yu Chen" <ja...@gmail.com> wrote:
> >
> > CAUTION: This email originated from outside of the organization.
> Do not click links or open attachments unless you can confirm the sender
> and know the content is safe.
> >
> >
> >
> > @Gyula say that add a config for this but not expose it on the
> rest
> > api is right
> > when user start up session cluster,we can config (eg
> > jobmanager.rest.api.submit.job.allow-reset-config = true/false)
> to
> > controller this behavior
> >
> > Now we have an existing cluster, the admin configures
> checkpointing
> > interval by this session cluster
> >
> > jobmanager.rest.api.submit.job.allow-reset-config = true/false
> >
> > - true : allow user reset new value config
> > - false(default) : not allow user set new config from ,throw exp
> to
> > tell user what properties is admin was set it
> >
> > However, there will be a problem in doing this. If the user
> writes
> > this Flink Config in jar by hardcoding, the highest priority
> must be
> > the code at this time, so the problem may still exist, and the
> user
> > cannot be prevented from this behavior.
> >
> > what do you think ?
> >
> > @Hong @Gyula @Teoh @Danny
> >
> >
> > Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
> > >
> > > +1 for the proposal.
> > >
> > > @Hong : I feel that the inverted override flag should be a
> cluster setting
> > > and not something the user can override at will. I fear that
> this might
> > > defeat the purpose of the feature itself.
> > > So I think we should add a config for this but not expose it
> on the rest api
> > >
> > > Gyula
> > >
> > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
> <li...@amazon.co.uk.invalid>
> > > wrote:
> > >
> > > > +1 to this FLIP.
> > > > This is very useful for teams building a Flink platform to
> run jobs from
> > > > an external user
> > > >
> > > > +1 on Danny's comment on adding a configuration to allow
> inverted order of
> > > > overrides.
> > > > However, might it be better to include an "override" toggle
> in the REST
> > > > API itself? That way we can change the flink configuration
> override
> > > > behaviour without restarting the Flink cluster. This would
> make sense if we
> > > > are thinking of a Session cluster and deploying multiple
> Flink jobs to the
> > > > same cluster.
> > > >
> > > > Regards,
> > > > Hong
> > > >
> > > >
> > > > On 18/08/2022, 10:46, "Danny Cranmer" <
> dannycranmer@apache.org> wrote:
> > > >
> > > > CAUTION: This email originated from outside of the
> organization. Do
> > > > not click links or open attachments unless you can confirm
> the sender and
> > > > know the content is safe.
> > > >
> > > >
> > > >
> > > > +1 thanks for driving this FLIP.
> > > >
> > > > We actually have an internally forked equivalent of
> this, which is on
> > > > our
> > > > list to try to upstream. Your proposal would *almost*
> work "off the
> > > > shelf"
> > > > for us. We have an inverted order or priority:
> > > > - Rest API / Flink CLI > User Code > Cluster Config
> > > >
> > > > This is because our end users do not submit their jobs
> directly, our
> > > > service does it on their behalf. For our usecase we do
> not want to
> > > > allow
> > > > users to override certain values we set, since some are
> managed from
> > > > our
> > > > service configuration, example: checkpointing interval.
> > > >
> > > > How would you feel about including a cluster
> configuration to invert
> > > > the
> > > > order? This could be decoupled to a follow-up FLIP.
> > > >
> > > > Thanks,
> > > > Danny Cranmer
> > > >
> > > >
> > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <
> jam.gzczy@gmail.com>
> > > > wrote:
> > > >
> > > > > This is a good suggestion, we can start this work
> after we finish the
> > > > > discussion and vote
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > > >
> > > > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > > > >
> > > > > > I've created a new ticket [FLINK-28973] Extending
> > > > /jars/:jarid/plan API
> > > > > to
> > > > > > support setting Flink configs - ASF JIRA (apache.org
> )<
> > > > > > https://issues.apache.org/jira/browse/FLINK-28973>.
> for the job
> > > > plan
> > > > > API.
> > > > > > Maybe we can create a new umbrella ticket for
> FLIP-256 and put
> > > > > FLINK-27060
> > > > > > & FLINK-28973 as subtasks. WDYT?
> > > > > >
> > > > > > Best,
> > > > > > Zhanghao Chen
> > > > > > ________________________________
> > > > > > From: zhengyu chen <ja...@gmail.com>
> > > > > > Sent: Monday, August 15, 2022 18:06
> > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic
> Parameter With
> > > > Flink Rest
> > > > > > Api
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > >
> > > > > > We would like to start a discussion thread on
> FLIP-256 Support Job
> > > > > Dynamic
> > > > > > Parameter With Flink Rest Api
> > > > > > <
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > > >
> > > > > > [1]
> > > > > >
> > > > > > After the user submits the jar package, running a
> job through
> > > > restapi
> > > > > > (/jars/:jarid/run) [2] can only pass in
> (allowNonRestoredState,
> > > > > > savepointPath, programArg, entry-class, parallelism)
> parameters,
> > > > which is
> > > > > > obvious with the diversification of job parameters
> (eg Checkpoint
> > > > > address)
> > > > > >
> > > > > > This solves the problem that the user can pass in
> other parameters
> > > > when
> > > > > > submitting a job, avoiding the user to define these
> job parameters
> > > > in the
> > > > > > code, resulting in the need to repackage the job for
> each
> > > > modification
> > > > > >
> > > > > > There was some interest from users [3] from a meetup
> and the
> > > > mailing
> > > > > list.
> > > > > > Looking forward to comments and feedback, thanks!
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > >
> > > > > >
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > > > >
> > > > > > [3]
> https://issues.apache.org/jira/browse/FLINK-27060
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best
> > > > > >
> > > > > > ConradJam
> > > > > >
> > > > >
> > > >
> > > >
> >
>
> --
> Best
>
> ConradJam
>
>
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Gyula Fóra <gy...@gmail.com>.
+1 for Chesnay's suggestion and taking a simple pragmatic approach.
Gyula
On Mon, Aug 29, 2022 at 3:41 PM Chesnay Schepler <ch...@apache.org> wrote:
> TBH I don't really get why we're discussion the prioritization of config
> options vs. what users have set in their application in this thread.
>
> Why don't we just follow the same order that the CLI currently enforces?
> Same point regarding the external vs internal parameters.
>
> Ultimately this FLIP is only about getting closer to achieving feature
> parity between the CLI and jar submission, so let's focus on that.
> Everything else should cover both the REST API and CLI, and would thus
> be out-of-scope of this FLIP imo.
>
> On 29/08/2022 03:52, Zheng Yu Chen wrote:
> > Hi Hong:
> >
> > I compared @gyula and yours solutions, here are some of my thoughts:
> >
> > We can't control whether the user defines some parameters in the code,
> so I
> > think User Code is always bigger than other configuration items (usually
> > code in development has the highest priority)
> > such as @gyula said, I think it's more reasonable to put User code
> > first. Then, according to the cluster configuration
> > (rest.submit.job.override-config = true/false), whether this job
> parameter
> > is configured to determine which value should take effect
> >
> > If we follow your strategy（Rest API / Flink CLI > User Code > Cluster
> > Config）, it means that after we submit the job, the user's hard-coded
> > configuration will be invalid, which will make the user feel strange: I
> > have configured this parameter in the code, why does it not take effect?
> > So I will be more inclined to support @gyula suggestion
> >
> > (Default) override true:
> > User code > rest api > cluster conf
> >
> > Override false :
> > User code > cluster conf > rest api
> >
> > @Hong What do you think ?
> >
> >
> >
> > Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月27日周六 16:06写道：
> >
> >> Hi Zheng Yu,
> >>
> >> Sorry for the late reply, I was on holiday last week.
> >>
> >> Could we propose the following config instead?
> >>
> >> rest.submit.job.override-config = true/false
> >> - true: REST API will have priority over user code (i.e. Rest API /
> >> Flink CLI > User Code > Cluster Config)
> >> - false (default): user code will have priority over REST API (i.e.
> User
> >> Code > Rest API / Flink CLI > Cluster Config)
> >>
> >> This way, the default behaviour will be according to the proposed FLIP,
> >> but we will have an additional toggle to ignore the configuration set in
> >> user code.
> >>
> >> > However, there will be a problem in doing this. If the user
> writes
> >> > this Flink Config in jar by hardcoding, the highest priority
> >> must be
> >> > the code at this time, so the problem may still exist, and
> the
> >> user
> >> > cannot be prevented from this behavior
> >>
> >> Hmm, would it be better if we set it such that the
> >> "rest.submit.job.override-config" cannot be overwritten in user code
> >> (ignored), just like configuration "jobmanager.rpc.port"?
> >>
> >> What do you think?
> >>
> >> Regards,
> >> Hong
> >>
> >>
> >>
> >> On 26/08/2022, 12:05, "Zheng Yu Chen" <ja...@gmail.com> wrote:
> >>
> >> CAUTION: This email originated from outside of the organization. Do
> >> not click links or open attachments unless you can confirm the sender
> and
> >> know the content is safe.
> >>
> >>
> >>
> >> HI Hong,
> >>
> >> Maybe you forgot. I don’t know if the current FLIP still needs to
> be
> >> modified. If not, we will start with the current plan. If you have
> >> relevant ideas in the future, please continue to discuss. If not,
> we
> >> will open voting at a later date.
> >>
> >> Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道：
> >> >
> >> > Hi Zheng Yu,
> >> >
> >> > We would have to take the same "cluster configuration" (cannot be
> >> set on job submission) vs "job configuration" approach in the User code
> as
> >> well. And we can classify
> jobmanager.rest.api.submit.job.allow-reset-config
> >> as a cluster configuration. That way, neither the REST API / User code
> can
> >> override this configuration, and it can only be set in cluster
> >> configuration on startup.
> >> >
> >> > There are other cluster specific configurations that are already
> >> treated this way (e.g. jobmanager.rpc.address,
> >> jobmanager.memory.process.size). As part of this work, I wonder if we
> can
> >> update the Flink docs on configuration (
> >>
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/
> )
> >> to specify which configuration is "cluster" and which is "job".
> >> >
> >> > Regards,
> >> > Hong
> >> >
> >> >
> >> >
> >> > On 20/08/2022, 09:16, "Zheng Yu Chen" <ja...@gmail.com>
> wrote:
> >> >
> >> > CAUTION: This email originated from outside of the
> organization.
> >> Do not click links or open attachments unless you can confirm the sender
> >> and know the content is safe.
> >> >
> >> >
> >> >
> >> > @Gyula say that add a config for this but not expose it on
> the
> >> rest
> >> > api is right
> >> > when user start up session cluster,we can config (eg
> >> > jobmanager.rest.api.submit.job.allow-reset-config =
> true/false)
> >> to
> >> > controller this behavior
> >> >
> >> > Now we have an existing cluster, the admin configures
> >> checkpointing
> >> > interval by this session cluster
> >> >
> >> > jobmanager.rest.api.submit.job.allow-reset-config =
> true/false
> >> >
> >> > - true : allow user reset new value config
> >> > - false(default) : not allow user set new config from ,throw
> exp
> >> to
> >> > tell user what properties is admin was set it
> >> >
> >> > However, there will be a problem in doing this. If the user
> >> writes
> >> > this Flink Config in jar by hardcoding, the highest priority
> >> must be
> >> > the code at this time, so the problem may still exist, and
> the
> >> user
> >> > cannot be prevented from this behavior.
> >> >
> >> > what do you think ?
> >> >
> >> > @Hong @Gyula @Teoh @Danny
> >> >
> >> >
> >> > Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
> >> > >
> >> > > +1 for the proposal.
> >> > >
> >> > > @Hong : I feel that the inverted override flag should be a
> >> cluster setting
> >> > > and not something the user can override at will. I fear
> that
> >> this might
> >> > > defeat the purpose of the feature itself.
> >> > > So I think we should add a config for this but not expose
> it
> >> on the rest api
> >> > >
> >> > > Gyula
> >> > >
> >> > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
> >> <li...@amazon.co.uk.invalid>
> >> > > wrote:
> >> > >
> >> > > > +1 to this FLIP.
> >> > > > This is very useful for teams building a Flink platform
> to
> >> run jobs from
> >> > > > an external user
> >> > > >
> >> > > > +1 on Danny's comment on adding a configuration to allow
> >> inverted order of
> >> > > > overrides.
> >> > > > However, might it be better to include an "override"
> toggle
> >> in the REST
> >> > > > API itself? That way we can change the flink
> configuration
> >> override
> >> > > > behaviour without restarting the Flink cluster. This
> would
> >> make sense if we
> >> > > > are thinking of a Session cluster and deploying multiple
> >> Flink jobs to the
> >> > > > same cluster.
> >> > > >
> >> > > > Regards,
> >> > > > Hong
> >> > > >
> >> > > >
> >> > > > On 18/08/2022, 10:46, "Danny Cranmer" <
> >> dannycranmer@apache.org> wrote:
> >> > > >
> >> > > > CAUTION: This email originated from outside of the
> >> organization. Do
> >> > > > not click links or open attachments unless you can
> confirm
> >> the sender and
> >> > > > know the content is safe.
> >> > > >
> >> > > >
> >> > > >
> >> > > > +1 thanks for driving this FLIP.
> >> > > >
> >> > > > We actually have an internally forked equivalent of
> >> this, which is on
> >> > > > our
> >> > > > list to try to upstream. Your proposal would *almost*
> >> work "off the
> >> > > > shelf"
> >> > > > for us. We have an inverted order or priority:
> >> > > > - Rest API / Flink CLI > User Code > Cluster Config
> >> > > >
> >> > > > This is because our end users do not submit their
> jobs
> >> directly, our
> >> > > > service does it on their behalf. For our usecase we
> do
> >> not want to
> >> > > > allow
> >> > > > users to override certain values we set, since some
> are
> >> managed from
> >> > > > our
> >> > > > service configuration, example: checkpointing
> interval.
> >> > > >
> >> > > > How would you feel about including a cluster
> >> configuration to invert
> >> > > > the
> >> > > > order? This could be decoupled to a follow-up FLIP.
> >> > > >
> >> > > > Thanks,
> >> > > > Danny Cranmer
> >> > > >
> >> > > >
> >> > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <
> >> jam.gzczy@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > This is a good suggestion, we can start this work
> >> after we finish the
> >> > > > > discussion and vote
> >> > > > >
> >> > > > > --
> >> > > > > Best
> >> > > > >
> >> > > > > ConradJam
> >> > > > >
> >> > > > >
> >> > > > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> >> > > > >
> >> > > > > > I've created a new ticket [FLINK-28973] Extending
> >> > > > /jars/:jarid/plan API
> >> > > > > to
> >> > > > > > support setting Flink configs - ASF JIRA (
> apache.org
> >> )<
> >> > > > > >
> https://issues.apache.org/jira/browse/FLINK-28973>.
> >> for the job
> >> > > > plan
> >> > > > > API.
> >> > > > > > Maybe we can create a new umbrella ticket for
> >> FLIP-256 and put
> >> > > > > FLINK-27060
> >> > > > > > & FLINK-28973 as subtasks. WDYT?
> >> > > > > >
> >> > > > > > Best,
> >> > > > > > Zhanghao Chen
> >> > > > > > ________________________________
> >> > > > > > From: zhengyu chen <ja...@gmail.com>
> >> > > > > > Sent: Monday, August 15, 2022 18:06
> >> > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> >> > > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic
> >> Parameter With
> >> > > > Flink Rest
> >> > > > > > Api
> >> > > > > >
> >> > > > > > Hi all,
> >> > > > > >
> >> > > > > >
> >> > > > > > We would like to start a discussion thread on
> >> FLIP-256 Support Job
> >> > > > > Dynamic
> >> > > > > > Parameter With Flink Rest Api
> >> > > > > > <
> >> > > > > >
> >> > > > >
> >> > > >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> >> > > > > > >
> >> > > > > > [1]
> >> > > > > >
> >> > > > > > After the user submits the jar package, running a
> >> job through
> >> > > > restapi
> >> > > > > > (/jars/:jarid/run) [2] can only pass in
> >> (allowNonRestoredState,
> >> > > > > > savepointPath, programArg, entry-class,
> parallelism)
> >> parameters,
> >> > > > which is
> >> > > > > > obvious with the diversification of job
> parameters
> >> (eg Checkpoint
> >> > > > > address)
> >> > > > > >
> >> > > > > > This solves the problem that the user can pass in
> >> other parameters
> >> > > > when
> >> > > > > > submitting a job, avoiding the user to define
> these
> >> job parameters
> >> > > > in the
> >> > > > > > code, resulting in the need to repackage the job
> for
> >> each
> >> > > > modification
> >> > > > > >
> >> > > > > > There was some interest from users [3] from a
> meetup
> >> and the
> >> > > > mailing
> >> > > > > list.
> >> > > > > > Looking forward to comments and feedback, thanks!
> >> > > > > >
> >> > > > > >
> >> > > > > > [1]
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> >> > > > > >
> >> > > > > >
> >> > > > > > [2]
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >>
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> >> > > > > >
> >> > > > > > [3]
> >> https://issues.apache.org/jira/browse/FLINK-27060
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Best
> >> > > > > >
> >> > > > > > ConradJam
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> >
> >>
> >> --
> >> Best
> >>
> >> ConradJam
> >>
> >>
>
>
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Chesnay Schepler <ch...@apache.org>.
TBH I don't really get why we're discussion the prioritization of config
options vs. what users have set in their application in this thread.
Why don't we just follow the same order that the CLI currently enforces?
Same point regarding the external vs internal parameters.
Ultimately this FLIP is only about getting closer to achieving feature
parity between the CLI and jar submission, so let's focus on that.
Everything else should cover both the REST API and CLI, and would thus
be out-of-scope of this FLIP imo.
On 29/08/2022 03:52, Zheng Yu Chen wrote:
> Hi Hong:
>
> I compared @gyula and yours solutions, here are some of my thoughts:
>
> We can't control whether the user defines some parameters in the code, so I
> think User Code is always bigger than other configuration items (usually
> code in development has the highest priority)
> such as @gyula said, I think it's more reasonable to put User code
> first. Then, according to the cluster configuration
> (rest.submit.job.override-config = true/false), whether this job parameter
> is configured to determine which value should take effect
>
> If we follow your strategy（Rest API / Flink CLI > User Code > Cluster
> Config）, it means that after we submit the job, the user's hard-coded
> configuration will be invalid, which will make the user feel strange: I
> have configured this parameter in the code, why does it not take effect?
> So I will be more inclined to support @gyula suggestion
>
> (Default) override true:
> User code > rest api > cluster conf
>
> Override false :
> User code > cluster conf > rest api
>
> @Hong What do you think ?
>
>
>
> Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月27日周六 16:06写道：
>
>> Hi Zheng Yu,
>>
>> Sorry for the late reply, I was on holiday last week.
>>
>> Could we propose the following config instead?
>>
>> rest.submit.job.override-config = true/false
>> - true: REST API will have priority over user code (i.e. Rest API /
>> Flink CLI > User Code > Cluster Config)
>> - false (default): user code will have priority over REST API (i.e. User
>> Code > Rest API / Flink CLI > Cluster Config)
>>
>> This way, the default behaviour will be according to the proposed FLIP,
>> but we will have an additional toggle to ignore the configuration set in
>> user code.
>>
>> > However, there will be a problem in doing this. If the user writes
>> > this Flink Config in jar by hardcoding, the highest priority
>> must be
>> > the code at this time, so the problem may still exist, and the
>> user
>> > cannot be prevented from this behavior
>>
>> Hmm, would it be better if we set it such that the
>> "rest.submit.job.override-config" cannot be overwritten in user code
>> (ignored), just like configuration "jobmanager.rpc.port"?
>>
>> What do you think?
>>
>> Regards,
>> Hong
>>
>>
>>
>> On 26/08/2022, 12:05, "Zheng Yu Chen" <ja...@gmail.com> wrote:
>>
>> CAUTION: This email originated from outside of the organization. Do
>> not click links or open attachments unless you can confirm the sender and
>> know the content is safe.
>>
>>
>>
>> HI Hong,
>>
>> Maybe you forgot. I don’t know if the current FLIP still needs to be
>> modified. If not, we will start with the current plan. If you have
>> relevant ideas in the future, please continue to discuss. If not, we
>> will open voting at a later date.
>>
>> Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道：
>> >
>> > Hi Zheng Yu,
>> >
>> > We would have to take the same "cluster configuration" (cannot be
>> set on job submission) vs "job configuration" approach in the User code as
>> well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config
>> as a cluster configuration. That way, neither the REST API / User code can
>> override this configuration, and it can only be set in cluster
>> configuration on startup.
>> >
>> > There are other cluster specific configurations that are already
>> treated this way (e.g. jobmanager.rpc.address,
>> jobmanager.memory.process.size). As part of this work, I wonder if we can
>> update the Flink docs on configuration (
>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/)
>> to specify which configuration is "cluster" and which is "job".
>> >
>> > Regards,
>> > Hong
>> >
>> >
>> >
>> > On 20/08/2022, 09:16, "Zheng Yu Chen" <ja...@gmail.com> wrote:
>> >
>> > CAUTION: This email originated from outside of the organization.
>> Do not click links or open attachments unless you can confirm the sender
>> and know the content is safe.
>> >
>> >
>> >
>> > @Gyula say that add a config for this but not expose it on the
>> rest
>> > api is right
>> > when user start up session cluster,we can config (eg
>> > jobmanager.rest.api.submit.job.allow-reset-config = true/false)
>> to
>> > controller this behavior
>> >
>> > Now we have an existing cluster, the admin configures
>> checkpointing
>> > interval by this session cluster
>> >
>> > jobmanager.rest.api.submit.job.allow-reset-config = true/false
>> >
>> > - true : allow user reset new value config
>> > - false(default) : not allow user set new config from ,throw exp
>> to
>> > tell user what properties is admin was set it
>> >
>> > However, there will be a problem in doing this. If the user
>> writes
>> > this Flink Config in jar by hardcoding, the highest priority
>> must be
>> > the code at this time, so the problem may still exist, and the
>> user
>> > cannot be prevented from this behavior.
>> >
>> > what do you think ?
>> >
>> > @Hong @Gyula @Teoh @Danny
>> >
>> >
>> > Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
>> > >
>> > > +1 for the proposal.
>> > >
>> > > @Hong : I feel that the inverted override flag should be a
>> cluster setting
>> > > and not something the user can override at will. I fear that
>> this might
>> > > defeat the purpose of the feature itself.
>> > > So I think we should add a config for this but not expose it
>> on the rest api
>> > >
>> > > Gyula
>> > >
>> > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
>> <li...@amazon.co.uk.invalid>
>> > > wrote:
>> > >
>> > > > +1 to this FLIP.
>> > > > This is very useful for teams building a Flink platform to
>> run jobs from
>> > > > an external user
>> > > >
>> > > > +1 on Danny's comment on adding a configuration to allow
>> inverted order of
>> > > > overrides.
>> > > > However, might it be better to include an "override" toggle
>> in the REST
>> > > > API itself? That way we can change the flink configuration
>> override
>> > > > behaviour without restarting the Flink cluster. This would
>> make sense if we
>> > > > are thinking of a Session cluster and deploying multiple
>> Flink jobs to the
>> > > > same cluster.
>> > > >
>> > > > Regards,
>> > > > Hong
>> > > >
>> > > >
>> > > > On 18/08/2022, 10:46, "Danny Cranmer" <
>> dannycranmer@apache.org> wrote:
>> > > >
>> > > > CAUTION: This email originated from outside of the
>> organization. Do
>> > > > not click links or open attachments unless you can confirm
>> the sender and
>> > > > know the content is safe.
>> > > >
>> > > >
>> > > >
>> > > > +1 thanks for driving this FLIP.
>> > > >
>> > > > We actually have an internally forked equivalent of
>> this, which is on
>> > > > our
>> > > > list to try to upstream. Your proposal would *almost*
>> work "off the
>> > > > shelf"
>> > > > for us. We have an inverted order or priority:
>> > > > - Rest API / Flink CLI > User Code > Cluster Config
>> > > >
>> > > > This is because our end users do not submit their jobs
>> directly, our
>> > > > service does it on their behalf. For our usecase we do
>> not want to
>> > > > allow
>> > > > users to override certain values we set, since some are
>> managed from
>> > > > our
>> > > > service configuration, example: checkpointing interval.
>> > > >
>> > > > How would you feel about including a cluster
>> configuration to invert
>> > > > the
>> > > > order? This could be decoupled to a follow-up FLIP.
>> > > >
>> > > > Thanks,
>> > > > Danny Cranmer
>> > > >
>> > > >
>> > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <
>> jam.gzczy@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > This is a good suggestion, we can start this work
>> after we finish the
>> > > > > discussion and vote
>> > > > >
>> > > > > --
>> > > > > Best
>> > > > >
>> > > > > ConradJam
>> > > > >
>> > > > >
>> > > > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
>> > > > >
>> > > > > > I've created a new ticket [FLINK-28973] Extending
>> > > > /jars/:jarid/plan API
>> > > > > to
>> > > > > > support setting Flink configs - ASF JIRA (apache.org
>> )<
>> > > > > > https://issues.apache.org/jira/browse/FLINK-28973>.
>> for the job
>> > > > plan
>> > > > > API.
>> > > > > > Maybe we can create a new umbrella ticket for
>> FLIP-256 and put
>> > > > > FLINK-27060
>> > > > > > & FLINK-28973 as subtasks. WDYT?
>> > > > > >
>> > > > > > Best,
>> > > > > > Zhanghao Chen
>> > > > > > ________________________________
>> > > > > > From: zhengyu chen <ja...@gmail.com>
>> > > > > > Sent: Monday, August 15, 2022 18:06
>> > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
>> > > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic
>> Parameter With
>> > > > Flink Rest
>> > > > > > Api
>> > > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > >
>> > > > > > We would like to start a discussion thread on
>> FLIP-256 Support Job
>> > > > > Dynamic
>> > > > > > Parameter With Flink Rest Api
>> > > > > > <
>> > > > > >
>> > > > >
>> > > >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
>> > > > > > >
>> > > > > > [1]
>> > > > > >
>> > > > > > After the user submits the jar package, running a
>> job through
>> > > > restapi
>> > > > > > (/jars/:jarid/run) [2] can only pass in
>> (allowNonRestoredState,
>> > > > > > savepointPath, programArg, entry-class, parallelism)
>> parameters,
>> > > > which is
>> > > > > > obvious with the diversification of job parameters
>> (eg Checkpoint
>> > > > > address)
>> > > > > >
>> > > > > > This solves the problem that the user can pass in
>> other parameters
>> > > > when
>> > > > > > submitting a job, avoiding the user to define these
>> job parameters
>> > > > in the
>> > > > > > code, resulting in the need to repackage the job for
>> each
>> > > > modification
>> > > > > >
>> > > > > > There was some interest from users [3] from a meetup
>> and the
>> > > > mailing
>> > > > > list.
>> > > > > > Looking forward to comments and feedback, thanks!
>> > > > > >
>> > > > > >
>> > > > > > [1]
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
>> > > > > >
>> > > > > >
>> > > > > > [2]
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
>> > > > > >
>> > > > > > [3]
>> https://issues.apache.org/jira/browse/FLINK-27060
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Best
>> > > > > >
>> > > > > > ConradJam
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> >
>>
>> --
>> Best
>>
>> ConradJam
>>
>>
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Zheng Yu Chen <ja...@gmail.com>.
Hi Hong:
I compared @gyula and yours solutions, here are some of my thoughts:
We can't control whether the user defines some parameters in the code, so I
think User Code is always bigger than other configuration items (usually
code in development has the highest priority)
such as @gyula said, I think it's more reasonable to put User code
first. Then, according to the cluster configuration
(rest.submit.job.override-config = true/false), whether this job parameter
is configured to determine which value should take effect
If we follow your strategy（Rest API / Flink CLI > User Code > Cluster
Config）, it means that after we submit the job, the user's hard-coded
configuration will be invalid, which will make the user feel strange: I
have configured this parameter in the code, why does it not take effect?
So I will be more inclined to support @gyula suggestion
(Default) override true:
User code > rest api > cluster conf
Override false :
User code > cluster conf > rest api
@Hong What do you think ?
Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月27日周六 16:06写道：
> Hi Zheng Yu,
>
> Sorry for the late reply, I was on holiday last week.
>
> Could we propose the following config instead?
>
> rest.submit.job.override-config = true/false
> - true: REST API will have priority over user code (i.e. Rest API /
> Flink CLI > User Code > Cluster Config)
> - false (default): user code will have priority over REST API (i.e. User
> Code > Rest API / Flink CLI > Cluster Config)
>
> This way, the default behaviour will be according to the proposed FLIP,
> but we will have an additional toggle to ignore the configuration set in
> user code.
>
> > However, there will be a problem in doing this. If the user writes
> > this Flink Config in jar by hardcoding, the highest priority
> must be
> > the code at this time, so the problem may still exist, and the
> user
> > cannot be prevented from this behavior
>
> Hmm, would it be better if we set it such that the
> "rest.submit.job.override-config" cannot be overwritten in user code
> (ignored), just like configuration "jobmanager.rpc.port"?
>
> What do you think?
>
> Regards,
> Hong
>
>
>
> On 26/08/2022, 12:05, "Zheng Yu Chen" <ja...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> HI Hong,
>
> Maybe you forgot. I don’t know if the current FLIP still needs to be
> modified. If not, we will start with the current plan. If you have
> relevant ideas in the future, please continue to discuss. If not, we
> will open voting at a later date.
>
> Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道：
> >
> > Hi Zheng Yu,
> >
> > We would have to take the same "cluster configuration" (cannot be
> set on job submission) vs "job configuration" approach in the User code as
> well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config
> as a cluster configuration. That way, neither the REST API / User code can
> override this configuration, and it can only be set in cluster
> configuration on startup.
> >
> > There are other cluster specific configurations that are already
> treated this way (e.g. jobmanager.rpc.address,
> jobmanager.memory.process.size). As part of this work, I wonder if we can
> update the Flink docs on configuration (
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/)
> to specify which configuration is "cluster" and which is "job".
> >
> > Regards,
> > Hong
> >
> >
> >
> > On 20/08/2022, 09:16, "Zheng Yu Chen" <ja...@gmail.com> wrote:
> >
> > CAUTION: This email originated from outside of the organization.
> Do not click links or open attachments unless you can confirm the sender
> and know the content is safe.
> >
> >
> >
> > @Gyula say that add a config for this but not expose it on the
> rest
> > api is right
> > when user start up session cluster,we can config (eg
> > jobmanager.rest.api.submit.job.allow-reset-config = true/false)
> to
> > controller this behavior
> >
> > Now we have an existing cluster, the admin configures
> checkpointing
> > interval by this session cluster
> >
> > jobmanager.rest.api.submit.job.allow-reset-config = true/false
> >
> > - true : allow user reset new value config
> > - false(default) : not allow user set new config from ,throw exp
> to
> > tell user what properties is admin was set it
> >
> > However, there will be a problem in doing this. If the user
> writes
> > this Flink Config in jar by hardcoding, the highest priority
> must be
> > the code at this time, so the problem may still exist, and the
> user
> > cannot be prevented from this behavior.
> >
> > what do you think ?
> >
> > @Hong @Gyula @Teoh @Danny
> >
> >
> > Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
> > >
> > > +1 for the proposal.
> > >
> > > @Hong : I feel that the inverted override flag should be a
> cluster setting
> > > and not something the user can override at will. I fear that
> this might
> > > defeat the purpose of the feature itself.
> > > So I think we should add a config for this but not expose it
> on the rest api
> > >
> > > Gyula
> > >
> > > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
> <li...@amazon.co.uk.invalid>
> > > wrote:
> > >
> > > > +1 to this FLIP.
> > > > This is very useful for teams building a Flink platform to
> run jobs from
> > > > an external user
> > > >
> > > > +1 on Danny's comment on adding a configuration to allow
> inverted order of
> > > > overrides.
> > > > However, might it be better to include an "override" toggle
> in the REST
> > > > API itself? That way we can change the flink configuration
> override
> > > > behaviour without restarting the Flink cluster. This would
> make sense if we
> > > > are thinking of a Session cluster and deploying multiple
> Flink jobs to the
> > > > same cluster.
> > > >
> > > > Regards,
> > > > Hong
> > > >
> > > >
> > > > On 18/08/2022, 10:46, "Danny Cranmer" <
> dannycranmer@apache.org> wrote:
> > > >
> > > > CAUTION: This email originated from outside of the
> organization. Do
> > > > not click links or open attachments unless you can confirm
> the sender and
> > > > know the content is safe.
> > > >
> > > >
> > > >
> > > > +1 thanks for driving this FLIP.
> > > >
> > > > We actually have an internally forked equivalent of
> this, which is on
> > > > our
> > > > list to try to upstream. Your proposal would *almost*
> work "off the
> > > > shelf"
> > > > for us. We have an inverted order or priority:
> > > > - Rest API / Flink CLI > User Code > Cluster Config
> > > >
> > > > This is because our end users do not submit their jobs
> directly, our
> > > > service does it on their behalf. For our usecase we do
> not want to
> > > > allow
> > > > users to override certain values we set, since some are
> managed from
> > > > our
> > > > service configuration, example: checkpointing interval.
> > > >
> > > > How would you feel about including a cluster
> configuration to invert
> > > > the
> > > > order? This could be decoupled to a follow-up FLIP.
> > > >
> > > > Thanks,
> > > > Danny Cranmer
> > > >
> > > >
> > > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <
> jam.gzczy@gmail.com>
> > > > wrote:
> > > >
> > > > > This is a good suggestion, we can start this work
> after we finish the
> > > > > discussion and vote
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > > >
> > > > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > > > >
> > > > > > I've created a new ticket [FLINK-28973] Extending
> > > > /jars/:jarid/plan API
> > > > > to
> > > > > > support setting Flink configs - ASF JIRA (apache.org
> )<
> > > > > > https://issues.apache.org/jira/browse/FLINK-28973>.
> for the job
> > > > plan
> > > > > API.
> > > > > > Maybe we can create a new umbrella ticket for
> FLIP-256 and put
> > > > > FLINK-27060
> > > > > > & FLINK-28973 as subtasks. WDYT?
> > > > > >
> > > > > > Best,
> > > > > > Zhanghao Chen
> > > > > > ________________________________
> > > > > > From: zhengyu chen <ja...@gmail.com>
> > > > > > Sent: Monday, August 15, 2022 18:06
> > > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic
> Parameter With
> > > > Flink Rest
> > > > > > Api
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > >
> > > > > > We would like to start a discussion thread on
> FLIP-256 Support Job
> > > > > Dynamic
> > > > > > Parameter With Flink Rest Api
> > > > > > <
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > > >
> > > > > > [1]
> > > > > >
> > > > > > After the user submits the jar package, running a
> job through
> > > > restapi
> > > > > > (/jars/:jarid/run) [2] can only pass in
> (allowNonRestoredState,
> > > > > > savepointPath, programArg, entry-class, parallelism)
> parameters,
> > > > which is
> > > > > > obvious with the diversification of job parameters
> (eg Checkpoint
> > > > > address)
> > > > > >
> > > > > > This solves the problem that the user can pass in
> other parameters
> > > > when
> > > > > > submitting a job, avoiding the user to define these
> job parameters
> > > > in the
> > > > > > code, resulting in the need to repackage the job for
> each
> > > > modification
> > > > > >
> > > > > > There was some interest from users [3] from a meetup
> and the
> > > > mailing
> > > > > list.
> > > > > > Looking forward to comments and feedback, thanks!
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > >
> > > > > >
> > > > > > [2]
> > > > > >
> > > > > >
> > > > >
> > > >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > > > >
> > > > > > [3]
> https://issues.apache.org/jira/browse/FLINK-27060
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best
> > > > > >
> > > > > > ConradJam
> > > > > >
> > > > >
> > > >
> > > >
> >
>
> --
> Best
>
> ConradJam
>
>
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by "Teoh, Hong" <li...@amazon.co.uk.INVALID>.
Hi Zheng Yu,
Sorry for the late reply, I was on holiday last week.
Could we propose the following config instead?
rest.submit.job.override-config = true/false
- true: REST API will have priority over user code (i.e. Rest API / Flink CLI > User Code > Cluster Config)
- false (default): user code will have priority over REST API (i.e. User Code > Rest API / Flink CLI > Cluster Config)
This way, the default behaviour will be according to the proposed FLIP, but we will have an additional toggle to ignore the configuration set in user code.
> However, there will be a problem in doing this. If the user writes
> this Flink Config in jar by hardcoding, the highest priority must be
> the code at this time, so the problem may still exist, and the user
> cannot be prevented from this behavior
Hmm, would it be better if we set it such that the "rest.submit.job.override-config" cannot be overwritten in user code (ignored), just like configuration "jobmanager.rpc.port"?
What do you think?
Regards,
Hong
On 26/08/2022, 12:05, "Zheng Yu Chen" <ja...@gmail.com> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
HI Hong,
Maybe you forgot. I don’t know if the current FLIP still needs to be
modified. If not, we will start with the current plan. If you have
relevant ideas in the future, please continue to discuss. If not, we
will open voting at a later date.
Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道：
>
> Hi Zheng Yu,
>
> We would have to take the same "cluster configuration" (cannot be set on job submission) vs "job configuration" approach in the User code as well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config as a cluster configuration. That way, neither the REST API / User code can override this configuration, and it can only be set in cluster configuration on startup.
>
> There are other cluster specific configurations that are already treated this way (e.g. jobmanager.rpc.address, jobmanager.memory.process.size). As part of this work, I wonder if we can update the Flink docs on configuration (https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/) to specify which configuration is "cluster" and which is "job".
>
> Regards,
> Hong
>
>
>
> On 20/08/2022, 09:16, "Zheng Yu Chen" <ja...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> @Gyula say that add a config for this but not expose it on the rest
> api is right
> when user start up session cluster,we can config (eg
> jobmanager.rest.api.submit.job.allow-reset-config = true/false) to
> controller this behavior
>
> Now we have an existing cluster, the admin configures checkpointing
> interval by this session cluster
>
> jobmanager.rest.api.submit.job.allow-reset-config = true/false
>
> - true : allow user reset new value config
> - false(default) : not allow user set new config from ,throw exp to
> tell user what properties is admin was set it
>
> However, there will be a problem in doing this. If the user writes
> this Flink Config in jar by hardcoding, the highest priority must be
> the code at this time, so the problem may still exist, and the user
> cannot be prevented from this behavior.
>
> what do you think ?
>
> @Hong @Gyula @Teoh @Danny
>
>
> Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
> >
> > +1 for the proposal.
> >
> > @Hong : I feel that the inverted override flag should be a cluster setting
> > and not something the user can override at will. I fear that this might
> > defeat the purpose of the feature itself.
> > So I think we should add a config for this but not expose it on the rest api
> >
> > Gyula
> >
> > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong <li...@amazon.co.uk.invalid>
> > wrote:
> >
> > > +1 to this FLIP.
> > > This is very useful for teams building a Flink platform to run jobs from
> > > an external user
> > >
> > > +1 on Danny's comment on adding a configuration to allow inverted order of
> > > overrides.
> > > However, might it be better to include an "override" toggle in the REST
> > > API itself? That way we can change the flink configuration override
> > > behaviour without restarting the Flink cluster. This would make sense if we
> > > are thinking of a Session cluster and deploying multiple Flink jobs to the
> > > same cluster.
> > >
> > > Regards,
> > > Hong
> > >
> > >
> > > On 18/08/2022, 10:46, "Danny Cranmer" <da...@apache.org> wrote:
> > >
> > > CAUTION: This email originated from outside of the organization. Do
> > > not click links or open attachments unless you can confirm the sender and
> > > know the content is safe.
> > >
> > >
> > >
> > > +1 thanks for driving this FLIP.
> > >
> > > We actually have an internally forked equivalent of this, which is on
> > > our
> > > list to try to upstream. Your proposal would *almost* work "off the
> > > shelf"
> > > for us. We have an inverted order or priority:
> > > - Rest API / Flink CLI > User Code > Cluster Config
> > >
> > > This is because our end users do not submit their jobs directly, our
> > > service does it on their behalf. For our usecase we do not want to
> > > allow
> > > users to override certain values we set, since some are managed from
> > > our
> > > service configuration, example: checkpointing interval.
> > >
> > > How would you feel about including a cluster configuration to invert
> > > the
> > > order? This could be decoupled to a follow-up FLIP.
> > >
> > > Thanks,
> > > Danny Cranmer
> > >
> > >
> > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <ja...@gmail.com>
> > > wrote:
> > >
> > > > This is a good suggestion, we can start this work after we finish the
> > > > discussion and vote
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > > >
> > > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > > >
> > > > > I've created a new ticket [FLINK-28973] Extending
> > > /jars/:jarid/plan API
> > > > to
> > > > > support setting Flink configs - ASF JIRA (apache.org)<
> > > > > https://issues.apache.org/jira/browse/FLINK-28973>. for the job
> > > plan
> > > > API.
> > > > > Maybe we can create a new umbrella ticket for FLIP-256 and put
> > > > FLINK-27060
> > > > > & FLINK-28973 as subtasks. WDYT?
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > ________________________________
> > > > > From: zhengyu chen <ja...@gmail.com>
> > > > > Sent: Monday, August 15, 2022 18:06
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With
> > > Flink Rest
> > > > > Api
> > > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > We would like to start a discussion thread on FLIP-256 Support Job
> > > > Dynamic
> > > > > Parameter With Flink Rest Api
> > > > > <
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > >
> > > > > [1]
> > > > >
> > > > > After the user submits the jar package, running a job through
> > > restapi
> > > > > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > > > > savepointPath, programArg, entry-class, parallelism) parameters,
> > > which is
> > > > > obvious with the diversification of job parameters (eg Checkpoint
> > > > address)
> > > > >
> > > > > This solves the problem that the user can pass in other parameters
> > > when
> > > > > submitting a job, avoiding the user to define these job parameters
> > > in the
> > > > > code, resulting in the need to repackage the job for each
> > > modification
> > > > >
> > > > > There was some interest from users [3] from a meetup and the
> > > mailing
> > > > list.
> > > > > Looking forward to comments and feedback, thanks!
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > >
> > > > >
> > > > > [2]
> > > > >
> > > > >
> > > >
> > > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > > >
> > > > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > >
> > >
> > >
>
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Zheng Yu Chen <ja...@gmail.com>.
HI Hong,
Maybe you forgot. I don’t know if the current FLIP still needs to be
modified. If not, we will start with the current plan. If you have
relevant ideas in the future, please continue to discuss. If not, we
will open voting at a later date.
Teoh, Hong <li...@amazon.co.uk.invalid> 于2022年8月22日周一 16:02写道：
>
> Hi Zheng Yu,
>
> We would have to take the same "cluster configuration" (cannot be set on job submission) vs "job configuration" approach in the User code as well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config as a cluster configuration. That way, neither the REST API / User code can override this configuration, and it can only be set in cluster configuration on startup.
>
> There are other cluster specific configurations that are already treated this way (e.g. jobmanager.rpc.address, jobmanager.memory.process.size). As part of this work, I wonder if we can update the Flink docs on configuration (https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/) to specify which configuration is "cluster" and which is "job".
>
> Regards,
> Hong
>
>
>
> On 20/08/2022, 09:16, "Zheng Yu Chen" <ja...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> @Gyula say that add a config for this but not expose it on the rest
> api is right
> when user start up session cluster,we can config (eg
> jobmanager.rest.api.submit.job.allow-reset-config = true/false) to
> controller this behavior
>
> Now we have an existing cluster, the admin configures checkpointing
> interval by this session cluster
>
> jobmanager.rest.api.submit.job.allow-reset-config = true/false
>
> - true : allow user reset new value config
> - false(default) : not allow user set new config from ,throw exp to
> tell user what properties is admin was set it
>
> However, there will be a problem in doing this. If the user writes
> this Flink Config in jar by hardcoding, the highest priority must be
> the code at this time, so the problem may still exist, and the user
> cannot be prevented from this behavior.
>
> what do you think ?
>
> @Hong @Gyula @Teoh @Danny
>
>
> Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
> >
> > +1 for the proposal.
> >
> > @Hong : I feel that the inverted override flag should be a cluster setting
> > and not something the user can override at will. I fear that this might
> > defeat the purpose of the feature itself.
> > So I think we should add a config for this but not expose it on the rest api
> >
> > Gyula
> >
> > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong <li...@amazon.co.uk.invalid>
> > wrote:
> >
> > > +1 to this FLIP.
> > > This is very useful for teams building a Flink platform to run jobs from
> > > an external user
> > >
> > > +1 on Danny's comment on adding a configuration to allow inverted order of
> > > overrides.
> > > However, might it be better to include an "override" toggle in the REST
> > > API itself? That way we can change the flink configuration override
> > > behaviour without restarting the Flink cluster. This would make sense if we
> > > are thinking of a Session cluster and deploying multiple Flink jobs to the
> > > same cluster.
> > >
> > > Regards,
> > > Hong
> > >
> > >
> > > On 18/08/2022, 10:46, "Danny Cranmer" <da...@apache.org> wrote:
> > >
> > > CAUTION: This email originated from outside of the organization. Do
> > > not click links or open attachments unless you can confirm the sender and
> > > know the content is safe.
> > >
> > >
> > >
> > > +1 thanks for driving this FLIP.
> > >
> > > We actually have an internally forked equivalent of this, which is on
> > > our
> > > list to try to upstream. Your proposal would *almost* work "off the
> > > shelf"
> > > for us. We have an inverted order or priority:
> > > - Rest API / Flink CLI > User Code > Cluster Config
> > >
> > > This is because our end users do not submit their jobs directly, our
> > > service does it on their behalf. For our usecase we do not want to
> > > allow
> > > users to override certain values we set, since some are managed from
> > > our
> > > service configuration, example: checkpointing interval.
> > >
> > > How would you feel about including a cluster configuration to invert
> > > the
> > > order? This could be decoupled to a follow-up FLIP.
> > >
> > > Thanks,
> > > Danny Cranmer
> > >
> > >
> > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <ja...@gmail.com>
> > > wrote:
> > >
> > > > This is a good suggestion, we can start this work after we finish the
> > > > discussion and vote
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > > >
> > > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > > >
> > > > > I've created a new ticket [FLINK-28973] Extending
> > > /jars/:jarid/plan API
> > > > to
> > > > > support setting Flink configs - ASF JIRA (apache.org)<
> > > > > https://issues.apache.org/jira/browse/FLINK-28973>. for the job
> > > plan
> > > > API.
> > > > > Maybe we can create a new umbrella ticket for FLIP-256 and put
> > > > FLINK-27060
> > > > > & FLINK-28973 as subtasks. WDYT?
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > ________________________________
> > > > > From: zhengyu chen <ja...@gmail.com>
> > > > > Sent: Monday, August 15, 2022 18:06
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With
> > > Flink Rest
> > > > > Api
> > > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > We would like to start a discussion thread on FLIP-256 Support Job
> > > > Dynamic
> > > > > Parameter With Flink Rest Api
> > > > > <
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > >
> > > > > [1]
> > > > >
> > > > > After the user submits the jar package, running a job through
> > > restapi
> > > > > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > > > > savepointPath, programArg, entry-class, parallelism) parameters,
> > > which is
> > > > > obvious with the diversification of job parameters (eg Checkpoint
> > > > address)
> > > > >
> > > > > This solves the problem that the user can pass in other parameters
> > > when
> > > > > submitting a job, avoiding the user to define these job parameters
> > > in the
> > > > > code, resulting in the need to repackage the job for each
> > > modification
> > > > >
> > > > > There was some interest from users [3] from a meetup and the
> > > mailing
> > > > list.
> > > > > Looking forward to comments and feedback, thanks!
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > >
> > > > >
> > > > > [2]
> > > > >
> > > > >
> > > >
> > > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > > >
> > > > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > >
> > >
> > >
>
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by "Teoh, Hong" <li...@amazon.co.uk.INVALID>.
Hi Zheng Yu,
We would have to take the same "cluster configuration" (cannot be set on job submission) vs "job configuration" approach in the User code as well. And we can classify jobmanager.rest.api.submit.job.allow-reset-config as a cluster configuration. That way, neither the REST API / User code can override this configuration, and it can only be set in cluster configuration on startup.
There are other cluster specific configurations that are already treated this way (e.g. jobmanager.rpc.address, jobmanager.memory.process.size). As part of this work, I wonder if we can update the Flink docs on configuration (https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/) to specify which configuration is "cluster" and which is "job".
Regards,
Hong
On 20/08/2022, 09:16, "Zheng Yu Chen" <ja...@gmail.com> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
@Gyula say that add a config for this but not expose it on the rest
api is right
when user start up session cluster,we can config (eg
jobmanager.rest.api.submit.job.allow-reset-config = true/false) to
controller this behavior
Now we have an existing cluster, the admin configures checkpointing
interval by this session cluster
jobmanager.rest.api.submit.job.allow-reset-config = true/false
- true : allow user reset new value config
- false(default) : not allow user set new config from ,throw exp to
tell user what properties is admin was set it
However, there will be a problem in doing this. If the user writes
this Flink Config in jar by hardcoding, the highest priority must be
the code at this time, so the problem may still exist, and the user
cannot be prevented from this behavior.
what do you think ?
@Hong @Gyula @Teoh @Danny
Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
>
> +1 for the proposal.
>
> @Hong : I feel that the inverted override flag should be a cluster setting
> and not something the user can override at will. I fear that this might
> defeat the purpose of the feature itself.
> So I think we should add a config for this but not expose it on the rest api
>
> Gyula
>
> On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong <li...@amazon.co.uk.invalid>
> wrote:
>
> > +1 to this FLIP.
> > This is very useful for teams building a Flink platform to run jobs from
> > an external user
> >
> > +1 on Danny's comment on adding a configuration to allow inverted order of
> > overrides.
> > However, might it be better to include an "override" toggle in the REST
> > API itself? That way we can change the flink configuration override
> > behaviour without restarting the Flink cluster. This would make sense if we
> > are thinking of a Session cluster and deploying multiple Flink jobs to the
> > same cluster.
> >
> > Regards,
> > Hong
> >
> >
> > On 18/08/2022, 10:46, "Danny Cranmer" <da...@apache.org> wrote:
> >
> > CAUTION: This email originated from outside of the organization. Do
> > not click links or open attachments unless you can confirm the sender and
> > know the content is safe.
> >
> >
> >
> > +1 thanks for driving this FLIP.
> >
> > We actually have an internally forked equivalent of this, which is on
> > our
> > list to try to upstream. Your proposal would *almost* work "off the
> > shelf"
> > for us. We have an inverted order or priority:
> > - Rest API / Flink CLI > User Code > Cluster Config
> >
> > This is because our end users do not submit their jobs directly, our
> > service does it on their behalf. For our usecase we do not want to
> > allow
> > users to override certain values we set, since some are managed from
> > our
> > service configuration, example: checkpointing interval.
> >
> > How would you feel about including a cluster configuration to invert
> > the
> > order? This could be decoupled to a follow-up FLIP.
> >
> > Thanks,
> > Danny Cranmer
> >
> >
> > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <ja...@gmail.com>
> > wrote:
> >
> > > This is a good suggestion, we can start this work after we finish the
> > > discussion and vote
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> > >
> > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > >
> > > > I've created a new ticket [FLINK-28973] Extending
> > /jars/:jarid/plan API
> > > to
> > > > support setting Flink configs - ASF JIRA (apache.org)<
> > > > https://issues.apache.org/jira/browse/FLINK-28973>. for the job
> > plan
> > > API.
> > > > Maybe we can create a new umbrella ticket for FLIP-256 and put
> > > FLINK-27060
> > > > & FLINK-28973 as subtasks. WDYT?
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > ________________________________
> > > > From: zhengyu chen <ja...@gmail.com>
> > > > Sent: Monday, August 15, 2022 18:06
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With
> > Flink Rest
> > > > Api
> > > >
> > > > Hi all,
> > > >
> > > >
> > > > We would like to start a discussion thread on FLIP-256 Support Job
> > > Dynamic
> > > > Parameter With Flink Rest Api
> > > > <
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > >
> > > > [1]
> > > >
> > > > After the user submits the jar package, running a job through
> > restapi
> > > > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > > > savepointPath, programArg, entry-class, parallelism) parameters,
> > which is
> > > > obvious with the diversification of job parameters (eg Checkpoint
> > > address)
> > > >
> > > > This solves the problem that the user can pass in other parameters
> > when
> > > > submitting a job, avoiding the user to define these job parameters
> > in the
> > > > code, resulting in the need to repackage the job for each
> > modification
> > > >
> > > > There was some interest from users [3] from a meetup and the
> > mailing
> > > list.
> > > > Looking forward to comments and feedback, thanks!
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > >
> > > >
> > > > [2]
> > > >
> > > >
> > >
> > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > >
> > > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > >
> >
> >
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Zheng Yu Chen <ja...@gmail.com>.
@Gyula say that add a config for this but not expose it on the rest
api is right
when user start up session cluster,we can config (eg
jobmanager.rest.api.submit.job.allow-reset-config = true/false) to
controller this behavior
Now we have an existing cluster, the admin configures checkpointing
interval by this session cluster
jobmanager.rest.api.submit.job.allow-reset-config = true/false
- true : allow user reset new value config
- false(default) : not allow user set new config from ,throw exp to
tell user what properties is admin was set it
However, there will be a problem in doing this. If the user writes
this Flink Config in jar by hardcoding, the highest priority must be
the code at this time, so the problem may still exist, and the user
cannot be prevented from this behavior.
what do you think ?
@Hong @Gyula @Teoh @Danny
Gyula Fóra <gy...@gmail.com> 于2022年8月18日周四 18:37写道：
>
> +1 for the proposal.
>
> @Hong : I feel that the inverted override flag should be a cluster setting
> and not something the user can override at will. I fear that this might
> defeat the purpose of the feature itself.
> So I think we should add a config for this but not expose it on the rest api
>
> Gyula
>
> On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong <li...@amazon.co.uk.invalid>
> wrote:
>
> > +1 to this FLIP.
> > This is very useful for teams building a Flink platform to run jobs from
> > an external user
> >
> > +1 on Danny's comment on adding a configuration to allow inverted order of
> > overrides.
> > However, might it be better to include an "override" toggle in the REST
> > API itself? That way we can change the flink configuration override
> > behaviour without restarting the Flink cluster. This would make sense if we
> > are thinking of a Session cluster and deploying multiple Flink jobs to the
> > same cluster.
> >
> > Regards,
> > Hong
> >
> >
> > On 18/08/2022, 10:46, "Danny Cranmer" <da...@apache.org> wrote:
> >
> > CAUTION: This email originated from outside of the organization. Do
> > not click links or open attachments unless you can confirm the sender and
> > know the content is safe.
> >
> >
> >
> > +1 thanks for driving this FLIP.
> >
> > We actually have an internally forked equivalent of this, which is on
> > our
> > list to try to upstream. Your proposal would *almost* work "off the
> > shelf"
> > for us. We have an inverted order or priority:
> > - Rest API / Flink CLI > User Code > Cluster Config
> >
> > This is because our end users do not submit their jobs directly, our
> > service does it on their behalf. For our usecase we do not want to
> > allow
> > users to override certain values we set, since some are managed from
> > our
> > service configuration, example: checkpointing interval.
> >
> > How would you feel about including a cluster configuration to invert
> > the
> > order? This could be decoupled to a follow-up FLIP.
> >
> > Thanks,
> > Danny Cranmer
> >
> >
> > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <ja...@gmail.com>
> > wrote:
> >
> > > This is a good suggestion, we can start this work after we finish the
> > > discussion and vote
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> > >
> > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > >
> > > > I've created a new ticket [FLINK-28973] Extending
> > /jars/:jarid/plan API
> > > to
> > > > support setting Flink configs - ASF JIRA (apache.org)<
> > > > https://issues.apache.org/jira/browse/FLINK-28973>. for the job
> > plan
> > > API.
> > > > Maybe we can create a new umbrella ticket for FLIP-256 and put
> > > FLINK-27060
> > > > & FLINK-28973 as subtasks. WDYT?
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > ________________________________
> > > > From: zhengyu chen <ja...@gmail.com>
> > > > Sent: Monday, August 15, 2022 18:06
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With
> > Flink Rest
> > > > Api
> > > >
> > > > Hi all,
> > > >
> > > >
> > > > We would like to start a discussion thread on FLIP-256 Support Job
> > > Dynamic
> > > > Parameter With Flink Rest Api
> > > > <
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > >
> > > > [1]
> > > >
> > > > After the user submits the jar package, running a job through
> > restapi
> > > > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > > > savepointPath, programArg, entry-class, parallelism) parameters,
> > which is
> > > > obvious with the diversification of job parameters (eg Checkpoint
> > > address)
> > > >
> > > > This solves the problem that the user can pass in other parameters
> > when
> > > > submitting a job, avoiding the user to define these job parameters
> > in the
> > > > code, resulting in the need to repackage the job for each
> > modification
> > > >
> > > > There was some interest from users [3] from a meetup and the
> > mailing
> > > list.
> > > > Looking forward to comments and feedback, thanks!
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > >
> > > >
> > > > [2]
> > > >
> > > >
> > >
> > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > >
> > > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > >
> >
> >
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Zheng Yu Chen <ja...@gmail.com>.
I don't think this is a good idea because it breaks the jar run api, this
is just a job submit parameter, now a non-job parameter
(overwriteFlinkConfiguration) is introduced. flinkConfiguration is just an
extra extension, users can keep their submit job by original way
Jing Ge <ji...@ververica.com> 于2022年8月19日周五 07:00写道：
> +1 for the FLIP. I like this FLIP. It is very useful.
>
> I'd like to ask one more question:
>
> "
>
> - The internal parameters passed in by the user contains the external
> parameter. At this point, we can see that the user configures the same
> two
> parameters in the *external parameter* and *internal parameters*, in
> this case the Outer parameters is valid ("allowNonRestoredState": true)
>
> "
>
> Should we enable one more feature to let users decide which parameter will
> win if there is a conflict between the external parameter and internal
> parameter? By default, like you described, the external parameter will win.
> Beyond that, users could pass an additional parameter to tell Flink that
> all parameters in flinkConfiguration have higher priorities and will
> overwrite the value of the external parameter with the same name.
>
> This additional parameter e.g. "overwriteExternalParameter" (more feasible
> name could be reconsidered) could be defined within the flinkConfiguration
> like:
> {
> "entryClass":
> "org.apache.flink.streaming.examples.statemachine.StateMachineExample",
> "parallelism": "1",
> "programArgs": null,
> "savepointPath": "hdfs://flink/jobs/stateMachineExample/savepoints",
> "allowNonRestoredState": true,
> "flinkConfiguration": {
> *"overwriteExternalParameter" : true, // default value is false for
> backward compatibility*
> "allowNonRestoredState": false,
> "state.checkpoints.dir":
> "hdfs://flink/jobs/stateMachineExample/checkpoints"
> }
> }
> Or as a new external parameter e.g. "overwriteFlinkConfiguration"
>
> {
> "entryClass":
> "org.apache.flink.streaming.examples.statemachine.StateMachineExample",
> "parallelism": "1",
> "programArgs": null,
> "savepointPath": "hdfs://flink/jobs/stateMachineExample/savepoints",
> "allowNonRestoredState": true,
> *"overwriteFlinkConfiguration" : false, // default value is true for
> backward compatibility*
> "flinkConfiguration": {
> "allowNonRestoredState": false,
> "state.checkpoints.dir":
> "hdfs://flink/jobs/stateMachineExample/checkpoints"
> }
> }
> In this way, we will give even more flexibility to users. WDYT?
>
> Best regards,
> Jing
>
>
>
>
>
>
>
>
> On Thu, Aug 18, 2022 at 12:52 PM Teoh, Hong <li...@amazon.co.uk.invalid>
> wrote:
>
> > @Gyula: I was thinking of the use case where the managed platform also
> > handles the job submission.
> >
> > Setup #1
> > External user : Creating jar
> > ------------------------------------
> > Managed platform : Submit job
> > Managed platform : Manage Flink cluster
> >
> > As compared to
> >
> > Setup #2
> > External user : Creating jar
> > External user : Submit job
> > ------------------------------------
> > Managed platform : Manage Flink cluster
> >
> >
> > But I take your point that a REST API toggle doesn't make sense at all in
> > Setup #2, so it would be more inclusive to include the cluster
> > configuration since that works for both setups
> >
> >
> > Regards,
> > Hong
> >
> >
> >
> > On 18/08/2022, 11:38, "Gyula Fóra" <gy...@gmail.com> wrote:
> >
> > CAUTION: This email originated from outside of the organization. Do
> > not click links or open attachments unless you can confirm the sender and
> > know the content is safe.
> >
> >
> >
> > +1 for the proposal.
> >
> > @Hong : I feel that the inverted override flag should be a cluster
> > setting
> > and not something the user can override at will. I fear that this
> might
> > defeat the purpose of the feature itself.
> > So I think we should add a config for this but not expose it on the
> > rest api
> >
> > Gyula
> >
> > On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
> > <li...@amazon.co.uk.invalid>
> > wrote:
> >
> > > +1 to this FLIP.
> > > This is very useful for teams building a Flink platform to run jobs
> > from
> > > an external user
> > >
> > > +1 on Danny's comment on adding a configuration to allow inverted
> > order of
> > > overrides.
> > > However, might it be better to include an "override" toggle in the
> > REST
> > > API itself? That way we can change the flink configuration override
> > > behaviour without restarting the Flink cluster. This would make
> > sense if we
> > > are thinking of a Session cluster and deploying multiple Flink jobs
> > to the
> > > same cluster.
> > >
> > > Regards,
> > > Hong
> > >
> > >
> > > On 18/08/2022, 10:46, "Danny Cranmer" <da...@apache.org>
> > wrote:
> > >
> > > CAUTION: This email originated from outside of the
> organization.
> > Do
> > > not click links or open attachments unless you can confirm the
> > sender and
> > > know the content is safe.
> > >
> > >
> > >
> > > +1 thanks for driving this FLIP.
> > >
> > > We actually have an internally forked equivalent of this, which
> > is on
> > > our
> > > list to try to upstream. Your proposal would *almost* work "off
> > the
> > > shelf"
> > > for us. We have an inverted order or priority:
> > > - Rest API / Flink CLI > User Code > Cluster Config
> > >
> > > This is because our end users do not submit their jobs
> directly,
> > our
> > > service does it on their behalf. For our usecase we do not want
> > to
> > > allow
> > > users to override certain values we set, since some are managed
> > from
> > > our
> > > service configuration, example: checkpointing interval.
> > >
> > > How would you feel about including a cluster configuration to
> > invert
> > > the
> > > order? This could be decoupled to a follow-up FLIP.
> > >
> > > Thanks,
> > > Danny Cranmer
> > >
> > >
> > > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <
> > jam.gzczy@gmail.com>
> > > wrote:
> > >
> > > > This is a good suggestion, we can start this work after we
> > finish the
> > > > discussion and vote
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > > >
> > > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > > >
> > > > > I've created a new ticket [FLINK-28973] Extending
> > > /jars/:jarid/plan API
> > > > to
> > > > > support setting Flink configs - ASF JIRA (apache.org)<
> > > > > https://issues.apache.org/jira/browse/FLINK-28973>. for
> the
> > job
> > > plan
> > > > API.
> > > > > Maybe we can create a new umbrella ticket for FLIP-256 and
> > put
> > > > FLINK-27060
> > > > > & FLINK-28973 as subtasks. WDYT?
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > ________________________________
> > > > > From: zhengyu chen <ja...@gmail.com>
> > > > > Sent: Monday, August 15, 2022 18:06
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter
> > With
> > > Flink Rest
> > > > > Api
> > > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > We would like to start a discussion thread on FLIP-256
> > Support Job
> > > > Dynamic
> > > > > Parameter With Flink Rest Api
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > > >
> > > > > [1]
> > > > >
> > > > > After the user submits the jar package, running a job
> through
> > > restapi
> > > > > (/jars/:jarid/run) [2] can only pass in
> > (allowNonRestoredState,
> > > > > savepointPath, programArg, entry-class, parallelism)
> > parameters,
> > > which is
> > > > > obvious with the diversification of job parameters (eg
> > Checkpoint
> > > > address)
> > > > >
> > > > > This solves the problem that the user can pass in other
> > parameters
> > > when
> > > > > submitting a job, avoiding the user to define these job
> > parameters
> > > in the
> > > > > code, resulting in the need to repackage the job for each
> > > modification
> > > > >
> > > > > There was some interest from users [3] from a meetup and
> the
> > > mailing
> > > > list.
> > > > > Looking forward to comments and feedback, thanks!
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > >
> > > > >
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > > >
> > > > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > >
> > >
> > >
> >
> >
>
--
Best
ConradJam
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Jing Ge <ji...@ververica.com>.
+1 for the FLIP. I like this FLIP. It is very useful.
I'd like to ask one more question:
"
- The internal parameters passed in by the user contains the external
parameter. At this point, we can see that the user configures the same two
parameters in the *external parameter* and *internal parameters*, in
this case the Outer parameters is valid ("allowNonRestoredState": true)
"
Should we enable one more feature to let users decide which parameter will
win if there is a conflict between the external parameter and internal
parameter? By default, like you described, the external parameter will win.
Beyond that, users could pass an additional parameter to tell Flink that
all parameters in flinkConfiguration have higher priorities and will
overwrite the value of the external parameter with the same name.
This additional parameter e.g. "overwriteExternalParameter" (more feasible
name could be reconsidered) could be defined within the flinkConfiguration
like:
{
"entryClass":
"org.apache.flink.streaming.examples.statemachine.StateMachineExample",
"parallelism": "1",
"programArgs": null,
"savepointPath": "hdfs://flink/jobs/stateMachineExample/savepoints",
"allowNonRestoredState": true,
"flinkConfiguration": {
*"overwriteExternalParameter" : true, // default value is false for
backward compatibility*
"allowNonRestoredState": false,
"state.checkpoints.dir":
"hdfs://flink/jobs/stateMachineExample/checkpoints"
}
}
Or as a new external parameter e.g. "overwriteFlinkConfiguration"
{
"entryClass":
"org.apache.flink.streaming.examples.statemachine.StateMachineExample",
"parallelism": "1",
"programArgs": null,
"savepointPath": "hdfs://flink/jobs/stateMachineExample/savepoints",
"allowNonRestoredState": true,
*"overwriteFlinkConfiguration" : false, // default value is true for
backward compatibility*
"flinkConfiguration": {
"allowNonRestoredState": false,
"state.checkpoints.dir":
"hdfs://flink/jobs/stateMachineExample/checkpoints"
}
}
In this way, we will give even more flexibility to users. WDYT?
Best regards,
Jing
On Thu, Aug 18, 2022 at 12:52 PM Teoh, Hong <li...@amazon.co.uk.invalid>
wrote:
> @Gyula: I was thinking of the use case where the managed platform also
> handles the job submission.
>
> Setup #1
> External user : Creating jar
> ------------------------------------
> Managed platform : Submit job
> Managed platform : Manage Flink cluster
>
> As compared to
>
> Setup #2
> External user : Creating jar
> External user : Submit job
> ------------------------------------
> Managed platform : Manage Flink cluster
>
>
> But I take your point that a REST API toggle doesn't make sense at all in
> Setup #2, so it would be more inclusive to include the cluster
> configuration since that works for both setups
>
>
> Regards,
> Hong
>
>
>
> On 18/08/2022, 11:38, "Gyula Fóra" <gy...@gmail.com> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> +1 for the proposal.
>
> @Hong : I feel that the inverted override flag should be a cluster
> setting
> and not something the user can override at will. I fear that this might
> defeat the purpose of the feature itself.
> So I think we should add a config for this but not expose it on the
> rest api
>
> Gyula
>
> On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong
> <li...@amazon.co.uk.invalid>
> wrote:
>
> > +1 to this FLIP.
> > This is very useful for teams building a Flink platform to run jobs
> from
> > an external user
> >
> > +1 on Danny's comment on adding a configuration to allow inverted
> order of
> > overrides.
> > However, might it be better to include an "override" toggle in the
> REST
> > API itself? That way we can change the flink configuration override
> > behaviour without restarting the Flink cluster. This would make
> sense if we
> > are thinking of a Session cluster and deploying multiple Flink jobs
> to the
> > same cluster.
> >
> > Regards,
> > Hong
> >
> >
> > On 18/08/2022, 10:46, "Danny Cranmer" <da...@apache.org>
> wrote:
> >
> > CAUTION: This email originated from outside of the organization.
> Do
> > not click links or open attachments unless you can confirm the
> sender and
> > know the content is safe.
> >
> >
> >
> > +1 thanks for driving this FLIP.
> >
> > We actually have an internally forked equivalent of this, which
> is on
> > our
> > list to try to upstream. Your proposal would *almost* work "off
> the
> > shelf"
> > for us. We have an inverted order or priority:
> > - Rest API / Flink CLI > User Code > Cluster Config
> >
> > This is because our end users do not submit their jobs directly,
> our
> > service does it on their behalf. For our usecase we do not want
> to
> > allow
> > users to override certain values we set, since some are managed
> from
> > our
> > service configuration, example: checkpointing interval.
> >
> > How would you feel about including a cluster configuration to
> invert
> > the
> > order? This could be decoupled to a follow-up FLIP.
> >
> > Thanks,
> > Danny Cranmer
> >
> >
> > On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <
> jam.gzczy@gmail.com>
> > wrote:
> >
> > > This is a good suggestion, we can start this work after we
> finish the
> > > discussion and vote
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> > >
> > > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> > >
> > > > I've created a new ticket [FLINK-28973] Extending
> > /jars/:jarid/plan API
> > > to
> > > > support setting Flink configs - ASF JIRA (apache.org)<
> > > > https://issues.apache.org/jira/browse/FLINK-28973>. for the
> job
> > plan
> > > API.
> > > > Maybe we can create a new umbrella ticket for FLIP-256 and
> put
> > > FLINK-27060
> > > > & FLINK-28973 as subtasks. WDYT?
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > ________________________________
> > > > From: zhengyu chen <ja...@gmail.com>
> > > > Sent: Monday, August 15, 2022 18:06
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter
> With
> > Flink Rest
> > > > Api
> > > >
> > > > Hi all,
> > > >
> > > >
> > > > We would like to start a discussion thread on FLIP-256
> Support Job
> > > Dynamic
> > > > Parameter With Flink Rest Api
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > > >
> > > > [1]
> > > >
> > > > After the user submits the jar package, running a job through
> > restapi
> > > > (/jars/:jarid/run) [2] can only pass in
> (allowNonRestoredState,
> > > > savepointPath, programArg, entry-class, parallelism)
> parameters,
> > which is
> > > > obvious with the diversification of job parameters (eg
> Checkpoint
> > > address)
> > > >
> > > > This solves the problem that the user can pass in other
> parameters
> > when
> > > > submitting a job, avoiding the user to define these job
> parameters
> > in the
> > > > code, resulting in the need to repackage the job for each
> > modification
> > > >
> > > > There was some interest from users [3] from a meetup and the
> > mailing
> > > list.
> > > > Looking forward to comments and feedback, thanks!
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > >
> > > >
> > > > [2]
> > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > > >
> > > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > >
> >
> >
>
>
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by "Teoh, Hong" <li...@amazon.co.uk.INVALID>.
@Gyula: I was thinking of the use case where the managed platform also handles the job submission.
Setup #1
External user : Creating jar
------------------------------------
Managed platform : Submit job
Managed platform : Manage Flink cluster
As compared to
Setup #2
External user : Creating jar
External user : Submit job
------------------------------------
Managed platform : Manage Flink cluster
But I take your point that a REST API toggle doesn't make sense at all in Setup #2, so it would be more inclusive to include the cluster configuration since that works for both setups
Regards,
Hong
On 18/08/2022, 11:38, "Gyula Fóra" <gy...@gmail.com> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
+1 for the proposal.
@Hong : I feel that the inverted override flag should be a cluster setting
and not something the user can override at will. I fear that this might
defeat the purpose of the feature itself.
So I think we should add a config for this but not expose it on the rest api
Gyula
On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong <li...@amazon.co.uk.invalid>
wrote:
> +1 to this FLIP.
> This is very useful for teams building a Flink platform to run jobs from
> an external user
>
> +1 on Danny's comment on adding a configuration to allow inverted order of
> overrides.
> However, might it be better to include an "override" toggle in the REST
> API itself? That way we can change the flink configuration override
> behaviour without restarting the Flink cluster. This would make sense if we
> are thinking of a Session cluster and deploying multiple Flink jobs to the
> same cluster.
>
> Regards,
> Hong
>
>
> On 18/08/2022, 10:46, "Danny Cranmer" <da...@apache.org> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> +1 thanks for driving this FLIP.
>
> We actually have an internally forked equivalent of this, which is on
> our
> list to try to upstream. Your proposal would *almost* work "off the
> shelf"
> for us. We have an inverted order or priority:
> - Rest API / Flink CLI > User Code > Cluster Config
>
> This is because our end users do not submit their jobs directly, our
> service does it on their behalf. For our usecase we do not want to
> allow
> users to override certain values we set, since some are managed from
> our
> service configuration, example: checkpointing interval.
>
> How would you feel about including a cluster configuration to invert
> the
> order? This could be decoupled to a follow-up FLIP.
>
> Thanks,
> Danny Cranmer
>
>
> On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <ja...@gmail.com>
> wrote:
>
> > This is a good suggestion, we can start this work after we finish the
> > discussion and vote
> >
> > --
> > Best
> >
> > ConradJam
> >
> >
> > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> >
> > > I've created a new ticket [FLINK-28973] Extending
> /jars/:jarid/plan API
> > to
> > > support setting Flink configs - ASF JIRA (apache.org)<
> > > https://issues.apache.org/jira/browse/FLINK-28973>. for the job
> plan
> > API.
> > > Maybe we can create a new umbrella ticket for FLIP-256 and put
> > FLINK-27060
> > > & FLINK-28973 as subtasks. WDYT?
> > >
> > > Best,
> > > Zhanghao Chen
> > > ________________________________
> > > From: zhengyu chen <ja...@gmail.com>
> > > Sent: Monday, August 15, 2022 18:06
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With
> Flink Rest
> > > Api
> > >
> > > Hi all,
> > >
> > >
> > > We would like to start a discussion thread on FLIP-256 Support Job
> > Dynamic
> > > Parameter With Flink Rest Api
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > >
> > > [1]
> > >
> > > After the user submits the jar package, running a job through
> restapi
> > > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > > savepointPath, programArg, entry-class, parallelism) parameters,
> which is
> > > obvious with the diversification of job parameters (eg Checkpoint
> > address)
> > >
> > > This solves the problem that the user can pass in other parameters
> when
> > > submitting a job, avoiding the user to define these job parameters
> in the
> > > code, resulting in the need to repackage the job for each
> modification
> > >
> > > There was some interest from users [3] from a meetup and the
> mailing
> > list.
> > > Looking forward to comments and feedback, thanks!
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > >
> > >
> > > [2]
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > >
> > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > >
> > >
> > >
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> >
>
>
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Gyula Fóra <gy...@gmail.com>.
+1 for the proposal.
@Hong : I feel that the inverted override flag should be a cluster setting
and not something the user can override at will. I fear that this might
defeat the purpose of the feature itself.
So I think we should add a config for this but not expose it on the rest api
Gyula
On Thu, Aug 18, 2022 at 12:33 PM Teoh, Hong <li...@amazon.co.uk.invalid>
wrote:
> +1 to this FLIP.
> This is very useful for teams building a Flink platform to run jobs from
> an external user
>
> +1 on Danny's comment on adding a configuration to allow inverted order of
> overrides.
> However, might it be better to include an "override" toggle in the REST
> API itself? That way we can change the flink configuration override
> behaviour without restarting the Flink cluster. This would make sense if we
> are thinking of a Session cluster and deploying multiple Flink jobs to the
> same cluster.
>
> Regards,
> Hong
>
>
> On 18/08/2022, 10:46, "Danny Cranmer" <da...@apache.org> wrote:
>
> CAUTION: This email originated from outside of the organization. Do
> not click links or open attachments unless you can confirm the sender and
> know the content is safe.
>
>
>
> +1 thanks for driving this FLIP.
>
> We actually have an internally forked equivalent of this, which is on
> our
> list to try to upstream. Your proposal would *almost* work "off the
> shelf"
> for us. We have an inverted order or priority:
> - Rest API / Flink CLI > User Code > Cluster Config
>
> This is because our end users do not submit their jobs directly, our
> service does it on their behalf. For our usecase we do not want to
> allow
> users to override certain values we set, since some are managed from
> our
> service configuration, example: checkpointing interval.
>
> How would you feel about including a cluster configuration to invert
> the
> order? This could be decoupled to a follow-up FLIP.
>
> Thanks,
> Danny Cranmer
>
>
> On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <ja...@gmail.com>
> wrote:
>
> > This is a good suggestion, we can start this work after we finish the
> > discussion and vote
> >
> > --
> > Best
> >
> > ConradJam
> >
> >
> > <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> >
> > > I've created a new ticket [FLINK-28973] Extending
> /jars/:jarid/plan API
> > to
> > > support setting Flink configs - ASF JIRA (apache.org)<
> > > https://issues.apache.org/jira/browse/FLINK-28973>. for the job
> plan
> > API.
> > > Maybe we can create a new umbrella ticket for FLIP-256 and put
> > FLINK-27060
> > > & FLINK-28973 as subtasks. WDYT?
> > >
> > > Best,
> > > Zhanghao Chen
> > > ________________________________
> > > From: zhengyu chen <ja...@gmail.com>
> > > Sent: Monday, August 15, 2022 18:06
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With
> Flink Rest
> > > Api
> > >
> > > Hi all,
> > >
> > >
> > > We would like to start a discussion thread on FLIP-256 Support Job
> > Dynamic
> > > Parameter With Flink Rest Api
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > > >
> > > [1]
> > >
> > > After the user submits the jar package, running a job through
> restapi
> > > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > > savepointPath, programArg, entry-class, parallelism) parameters,
> which is
> > > obvious with the diversification of job parameters (eg Checkpoint
> > address)
> > >
> > > This solves the problem that the user can pass in other parameters
> when
> > > submitting a job, avoiding the user to define these job parameters
> in the
> > > code, resulting in the need to repackage the job for each
> modification
> > >
> > > There was some interest from users [3] from a meetup and the
> mailing
> > list.
> > > Looking forward to comments and feedback, thanks!
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > >
> > >
> > > [2]
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> > >
> > > [3] https://issues.apache.org/jira/browse/FLINK-27060
> > >
> > >
> > >
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> >
>
>
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by "Teoh, Hong" <li...@amazon.co.uk.INVALID>.
+1 to this FLIP.
This is very useful for teams building a Flink platform to run jobs from an external user
+1 on Danny's comment on adding a configuration to allow inverted order of overrides.
However, might it be better to include an "override" toggle in the REST API itself? That way we can change the flink configuration override behaviour without restarting the Flink cluster. This would make sense if we are thinking of a Session cluster and deploying multiple Flink jobs to the same cluster.
Regards,
Hong
On 18/08/2022, 10:46, "Danny Cranmer" <da...@apache.org> wrote:
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
+1 thanks for driving this FLIP.
We actually have an internally forked equivalent of this, which is on our
list to try to upstream. Your proposal would *almost* work "off the shelf"
for us. We have an inverted order or priority:
- Rest API / Flink CLI > User Code > Cluster Config
This is because our end users do not submit their jobs directly, our
service does it on their behalf. For our usecase we do not want to allow
users to override certain values we set, since some are managed from our
service configuration, example: checkpointing interval.
How would you feel about including a cluster configuration to invert the
order? This could be decoupled to a follow-up FLIP.
Thanks,
Danny Cranmer
On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <ja...@gmail.com> wrote:
> This is a good suggestion, we can start this work after we finish the
> discussion and vote
>
> --
> Best
>
> ConradJam
>
>
> <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
>
> > I've created a new ticket [FLINK-28973] Extending /jars/:jarid/plan API
> to
> > support setting Flink configs - ASF JIRA (apache.org)<
> > https://issues.apache.org/jira/browse/FLINK-28973>. for the job plan
> API.
> > Maybe we can create a new umbrella ticket for FLIP-256 and put
> FLINK-27060
> > & FLINK-28973 as subtasks. WDYT?
> >
> > Best,
> > Zhanghao Chen
> > ________________________________
> > From: zhengyu chen <ja...@gmail.com>
> > Sent: Monday, August 15, 2022 18:06
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest
> > Api
> >
> > Hi all,
> >
> >
> > We would like to start a discussion thread on FLIP-256 Support Job
> Dynamic
> > Parameter With Flink Rest Api
> > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > >
> > [1]
> >
> > After the user submits the jar package, running a job through restapi
> > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > savepointPath, programArg, entry-class, parallelism) parameters, which is
> > obvious with the diversification of job parameters (eg Checkpoint
> address)
> >
> > This solves the problem that the user can pass in other parameters when
> > submitting a job, avoiding the user to define these job parameters in the
> > code, resulting in the need to repackage the job for each modification
> >
> > There was some interest from users [3] from a meetup and the mailing
> list.
> > Looking forward to comments and feedback, thanks!
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> >
> >
> > [2]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> >
> > [3] https://issues.apache.org/jira/browse/FLINK-27060
> >
> >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Danny Cranmer <da...@apache.org>.
+1 thanks for driving this FLIP.
We actually have an internally forked equivalent of this, which is on our
list to try to upstream. Your proposal would *almost* work "off the shelf"
for us. We have an inverted order or priority:
- Rest API / Flink CLI > User Code > Cluster Config
This is because our end users do not submit their jobs directly, our
service does it on their behalf. For our usecase we do not want to allow
users to override certain values we set, since some are managed from our
service configuration, example: checkpointing interval.
How would you feel about including a cluster configuration to invert the
order? This could be decoupled to a follow-up FLIP.
Thanks,
Danny Cranmer
On Tue, Aug 16, 2022 at 7:21 AM Zheng Yu Chen <ja...@gmail.com> wrote:
> This is a good suggestion, we can start this work after we finish the
> discussion and vote
>
> --
> Best
>
> ConradJam
>
>
> <zh...@outlook.com> 于2022年8月15日周一 21:01写道：
>
> > I've created a new ticket [FLINK-28973] Extending /jars/:jarid/plan API
> to
> > support setting Flink configs - ASF JIRA (apache.org)<
> > https://issues.apache.org/jira/browse/FLINK-28973>. for the job plan
> API.
> > Maybe we can create a new umbrella ticket for FLIP-256 and put
> FLINK-27060
> > & FLINK-28973 as subtasks. WDYT?
> >
> > Best,
> > Zhanghao Chen
> > ________________________________
> > From: zhengyu chen <ja...@gmail.com>
> > Sent: Monday, August 15, 2022 18:06
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest
> > Api
> >
> > Hi all,
> >
> >
> > We would like to start a discussion thread on FLIP-256 Support Job
> Dynamic
> > Parameter With Flink Rest Api
> > <
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> > >
> > [1]
> >
> > After the user submits the jar package, running a job through restapi
> > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > savepointPath, programArg, entry-class, parallelism) parameters, which is
> > obvious with the diversification of job parameters (eg Checkpoint
> address)
> >
> > This solves the problem that the user can pass in other parameters when
> > submitting a job, avoiding the user to define these job parameters in the
> > code, resulting in the need to repackage the job for each modification
> >
> > There was some interest from users [3] from a meetup and the mailing
> list.
> > Looking forward to comments and feedback, thanks!
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> >
> >
> > [2]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
> >
> > [3] https://issues.apache.org/jira/browse/FLINK-27060
> >
> >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by Zheng Yu Chen <ja...@gmail.com>.
This is a good suggestion, we can start this work after we finish the
discussion and vote
--
Best
ConradJam
<zh...@outlook.com> 于2022年8月15日周一 21:01写道：
> I've created a new ticket [FLINK-28973] Extending /jars/:jarid/plan API to
> support setting Flink configs - ASF JIRA (apache.org)<
> https://issues.apache.org/jira/browse/FLINK-28973>. for the job plan API.
> Maybe we can create a new umbrella ticket for FLIP-256 and put FLINK-27060
> & FLINK-28973 as subtasks. WDYT?
>
> Best,
> Zhanghao Chen
> ________________________________
> From: zhengyu chen <ja...@gmail.com>
> Sent: Monday, August 15, 2022 18:06
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest
> Api
>
> Hi all,
>
>
> We would like to start a discussion thread on FLIP-256 Support Job Dynamic
> Parameter With Flink Rest Api
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> >
> [1]
>
> After the user submits the jar package, running a job through restapi
> (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> savepointPath, programArg, entry-class, parallelism) parameters, which is
> obvious with the diversification of job parameters (eg Checkpoint address)
>
> This solves the problem that the user can pass in other parameters when
> submitting a job, avoiding the user to define these job parameters in the
> code, resulting in the need to repackage the job for each modification
>
> There was some interest from users [3] from a meetup and the mailing list.
> Looking forward to comments and feedback, thanks!
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
>
>
> [2]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
>
> [3] https://issues.apache.org/jira/browse/FLINK-27060
>
>
>
>
> --
> Best
>
> ConradJam
>
Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by zh...@outlook.com.
I've created a new ticket [FLINK-28973] Extending /jars/:jarid/plan API to support setting Flink configs - ASF JIRA (apache.org)<https://issues.apache.org/jira/browse/FLINK-28973>. for the job plan API. Maybe we can create a new umbrella ticket for FLIP-256 and put FLINK-27060 & FLINK-28973 as subtasks. WDYT?
Best,
Zhanghao Chen
________________________________
From: zhengyu chen <ja...@gmail.com>
Sent: Monday, August 15, 2022 18:06
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Hi all,
We would like to start a discussion thread on FLIP-256 Support Job Dynamic
Parameter With Flink Rest Api
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api>
[1]
After the user submits the jar package, running a job through restapi
(/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
savepointPath, programArg, entry-class, parallelism) parameters, which is
obvious with the diversification of job parameters (eg Checkpoint address)
This solves the problem that the user can pass in other parameters when
submitting a job, avoiding the user to define these job parameters in the
code, resulting in the need to repackage the job for each modification
There was some interest from users [3] from a meetup and the mailing list.
Looking forward to comments and feedback, thanks!
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
[3] https://issues.apache.org/jira/browse/FLINK-27060
--
Best
ConradJam
Re:[DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api
Posted by JasonLee <17...@163.com>.
+1
Thanks for driving this,It makes the REST API that runs the job more powerful and complete
Best
JasonLee
---- Replied Message ----
| From | zhengyu chen<ja...@gmail.com> |
| Date | 08/15/2022 18:06 |
| To | <de...@flink.apache.org> |
| Subject | [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api |
Hi all,
We would like to start a discussion thread on FLIP-256 Support Job Dynamic
Parameter With Flink Rest Api
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api>
[1]
After the user submits the jar package, running a job through restapi
(/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
savepointPath, programArg, entry-class, parallelism) parameters, which is
obvious with the diversification of job parameters (eg Checkpoint address)
This solves the problem that the user can pass in other parameters when
submitting a job, avoiding the user to define these job parameters in the
code, resulting in the need to repackage the job for each modification
There was some interest from users [3] from a meetup and the mailing list.
Looking forward to comments and feedback, thanks!
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/rest_api/#jars-jarid-run
[3] https://issues.apache.org/jira/browse/FLINK-27060
--
Best
ConradJam