[jira] [Comment Edited] (FLINK-26570) Remote module configuration interpolation

["Fil Karnicki (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17504767#comment-17504767 ] 

Fil Karnicki edited comment on FLINK-26570 at 3/11/22, 7:48 AM:
----------------------------------------------------------------

Hi [~trohrmann] sounds like a good idea to me.

I recently opened a pull request, which among other things, uses job entrypoint args in the creation of the streaming execution environment for statefun jobs. This was so that jobs running from an uber jar could set various settings (like checkpointing interval) per job [1].  I wonder if we should include system and env properties there too (either before both the env and StatefulFunctionsConfig get created, or maybe after the env gets created, but before StatefulFunctionsConfig does). Having those settings be part of StatefulFunctionsConfig's globalConfig would automatically allow for the string interpolation inside RemodeModule#bindComponent.

I'm not so familiar with the project as to be able to foresee if loading system+env properties into StatefulFunctionsConfig isn't going to cause problems for existing users, but I hope Igal is going shed some light on this as part of the PR review.

[https://github.com/apache/flink-statefun/pull/307/files#diff-6249db934ffac5890c899612fc2daa31e7e85768faee779f1ccf17ddbbee23f7]

 

[1] - I know that the preferred way of launching statefun is via docker, but we need to make do with a shared cloudera cluster.


was (Author: JIRAUSER284249):
Hi [~trohrmann] sounds like a good idea to me.

I recently opened a pull request to use job entrypoint args in the creation of the streaming execution environment for statefun jobs. This was so that jobs running from an uber jar could set various settings (like checkpointing interval) per job [1].  I wonder if we should include system and env properties there too (either before both the env and StatefulFunctionsConfig get created, or maybe after the env gets created, but before StatefulFunctionsConfig does). Having those settings be part of StatefulFunctionsConfig's globalConfig would automatically allow for the string interpolation inside RemodeModule#bindComponent.

I'm not so familiar with the project as to be able to foresee if loading system+env properties into StatefulFunctionsConfig isn't going to cause problems for existing users, but I hope Igal is going shed some light on this as part of the PR review.

[https://github.com/apache/flink-statefun/pull/307/files#diff-6249db934ffac5890c899612fc2daa31e7e85768faee779f1ccf17ddbbee23f7]

 

[1] - I know that the preferred way of launching statefun is via docker, but we need to make do with a shared cloudera cluster.

> Remote module configuration interpolation
> -----------------------------------------
>
>                 Key: FLINK-26570
>                 URL: https://issues.apache.org/jira/browse/FLINK-26570
>             Project: Flink
>          Issue Type: Improvement
>          Components: Stateful Functions
>            Reporter: Fil Karnicki
>            Priority: Major
>
> Add the ability for users to provide placeholders in module.yaml, e.g.
> {code:java}
> kind: com.foo.bar/test
> spec:
>   something: ${REPLACE_ME}
>   transport:
>     password: ${REPLACE_ME_WITH_A_SECRET}
>     array:
>       - ${REPLACE_ME}
>       - sthElse {code}
> These placeholders would be resolved in 
> org.apache.flink.statefun.flink.core.jsonmodule.RemoteModule#bindComponent
> using 
> {code:java}
> ParameterTool.fromSystemProperties().mergeWith(ParameterTool.fromMap(globalConfiguration) {code}
> by traversing the ComponentJsonObject.specJsonNode() and replacing values that contain placeholders with values from the combined system+globalConfig map



--
This message was sent by Atlassian Jira
(v8.20.1#820001)