You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampark.apache.org by 田向阳 <lu...@163.com> on 2022/10/08 05:13:38 UTC
回复：[DISCUSS][issue-1477] Add variable management module
is this module include a resource management cente r where jars can be chosen to provide to create a job
---- 回复的原邮件 ----
| 发件人 | Huajie Wang<be...@gmail.com> |
| 日期 | 2022年10月08日 12:07 |
| 收件人 | dev@streampark.apache.org<de...@streampark.apache.org> |
| 抄送至 | |
| 主题 | Re: [DISCUSS][issue-1477] Add variable management module |
hi 阿穆:
Thank you for the clarification, My question is answered, but now I have
one more concern,
I'm currently using {var} for variables, and I'm worried that one of the
user's parameters includes "{" and "}", e.g :
--password 'mypass{123}abc'
So how do we parse parse this? How about ${var}
Best,
Huajie Wang
阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> Hi, monster, Huajie Wang, Rui Fan:
>
> Thanks for your reply.
>
> I have integrated your replies. I really don't pay much attention to SQL,
> so I will do the following.
>
> 1.Each team has independent variables that do not affect each other and
> are isolated from each other, but the same variable is unique under the
> same team, so you need to specify the team when creating variables (users
> may belong to multiple teams, and the teams of the current user will be
> displayed here for users to choose)
> 2.For Flink jar jobs, there are two ways to pass variables
> - By using placeholders, such as adding --kafka {mykafka} to the args of
> the application, 'mykafka' is the variable code of a variable
> - Search by variable code or variable name. After a variable is selected,
> the parameter transfer method is displayed
> 3.Flink sql will use variables as placeholders, E.g:
>
> CREATE TABLE KafkaTableA (
> `user_id` BIGINT,
> `item_id` BIGINT,
> `type` STRING
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'user_behaviorA',
> 'properties.bootstrap.servers' = '{kafka_clusterA}', --
> 'kafka_ ClusterA' is the variable code
> 'properties.group.id' = 'testGroupA',
> 'scan.startup.mode' = 'earliest-offset',
>
> CREATE TABLE KafkaTableB (
> `user_id` BIGINT,
> `item_id` BIGINT
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'user_behaviorB',
> 'properties.bootstrap.servers' = '{kafka_clusterB}', --
> 'kafka_ ClusterB' is the variable code
> 'properties.group.id' = 'testGroupB',
> 'scan.startup.mode' = 'earliest-offset',
>
> INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM
> KafkaTableB -- 'type' is the variable code
>
> 4.In the future, we can consider setting some system built-in variables
>
> Best,
> ChunJin Mu
>
> On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <1996fanrui@gmail.com> wrote:
>
> > Hi 阿穆:
> >
> > It's a great proposal.
> >
> > For usage scenarios, I strongly recommend support for replacing
> variables
> > in sql. E.g:
> >
> > CREATE TABLE KafkaTable (
> > `user_id` BIGINT,
> > `item_id` BIGINT
> > ) WITH (
> > 'connector' = 'kafka',
> > 'topic' = 'user_behavior',
> > 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > 'properties.group.id' = 'testGroup',
> > 'scan.startup.mode' = 'earliest-offset',
> > 'format' = 'csv'
> > )
> >
> > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
> >
> > For variable management, I have some question:
> >
> > 1. Is the parameter used within the team? If all teams use a common
> > parameter, it
> > needs to be defined for each team, right?
> > 2. Which users can modify variables? Can all users modify all
> variables of
> > their team?
> > After the variable is modified incorrectly, many jobs may be
> affected. So I
> > suggest that
> > only team admin or system admin have permission to modify, add and
> delete
> > variables.
> >
> > Best
> > fanrui
>
>
> On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <benjobx@gmail.com> wrote:
>
> > hi 阿穆:
> >
> >
> > Thanks for starting a very useful discussion, I have a some questions:
> >
> >
> > 1. The access boundary problem of variable management, is this all
> > people in the platform can access these variables( no matter who
> > defines) or that only team members can be access these variables.
> >
> > 2. There are two usage scenarios for variables that I know of, one is
> > in the entry parameter of main, and the other is in flinksql. Are both
> > supported, or only supported main entry ?
> >
> > about first question, my opinion:
> >
> > 1. The variable management is associated with the team. After the user
> > logs in to the system, he must select a team (a user can belong to one
> > or more teams), The variable belongs to a team
> >
> > 2. All resources between teams are isolated, which means that
> > variables are also isolated. In short, members of a team can use and
> > define these variables, which are invisible to other members, and
> > other members cannot use these variables.
> >
> >
> >
> > Welcome to your views
> >
> >
> > Best,
> > Huajie Wang
> >
> >
> >
> >
> > 陈卓宇 <monster.zhuoyuchen@gmail.com> 于2022年10月7日周五 14:18写道：
>
> > > Is it possible to do this variable support in sql as well
> > >
> > > 阿穆 <305343894@qq.com.invalid> 于2022年10月7日周五 13:54写道：
> > >
> > > > Hi guys,
> > > >
> > > >
> > > > I want to start a discussion about the variable management
> module. Please
> > > > see this issue issue-1477.
> > > >
> > > >
> > > > Background:
> > > >
> > > >
> > > > In the actual production environment, Flink jobs are
> generally complex,
> > > > and usually require multiple external components. For
> example, Flink jobs
> > > > consume data from Kafka, then connect external components
> such as HBase
> > > or
> > > > Redis to obtain additional business information, and then
> write it to the
> > > > downstream external components. There are the following
> problems:
> > > >
> > > >
> > > > 1. The connection information of external components, such
> as IP, port
> > > and
> > > > user password, needs to be configured in the application
> args and
> > > > transferred to the Flink job, so that the connection
> information of
> > > > external components is distributed in multiple
> applications. Once the
> > > > connection information of external components changes, many
> application
> > > > args parameters need to be modified, which will lead to
> high operation
> > > and
> > > > maintenance costs.
> > > > 2. There are many people in the team developing real-time
> computing jobs.
> > > > There is no uniform specification for the connection
> information of the
> > > > external components passed to the job, resulting in
> different parameter
> > > > names of the same component. This is difficult to count
> which external
> > > > components are dependent.
> > > > 3. In production practice, there are usually multiple sets
> of
> > > > environments, such as test environment and production
> environment. It is
> > > > not intuitive to judge whether a component belongs to a
> test environment
> > > or
> > > > a production environment simply through IP and ports. Even
> if it can be
> > > > judged, there are some omissions. In this way, the
> connection information
> > > > online to the production environment may be external
> components of the
> > > test
> > > > environment, or components of the production environment are
> > > inadvertently
> > > > configured during testing, leading to production failures.
> > > >
> > > >
> > > > Proposal:
> > > >
> > > >
> > > > 1. Create a unified module to manage variables. Users can
> define their
> > > own
> > > > variables in this module. Variable information includes
> variable code
> > > > (passed to Flink program as parameter name), variable
> value, variable
> > name,
> > > > variable description, and which applications depend on
> > > > 2. In the same Team module, the variable code or variable
> name is unique
> > > > - By using placeholders, such as adding -- kafka {mykafka}
> to the args of
> > > > the application, mykafka is the variable code of a variable
> > > > - Search by variable code or variable name. After a
> variable is selected,
> > > > the parameter transfer method is displayed
> > > >
> > > >
> > > >
> > > > If you have any suggestions, welcome to participate in the
> discussion on
> > > > the mail list or issue, I look forward to your feedback.
> > > >
> > > >
> > > > Best wishes
> > >
Re: [DISCUSS][issue-1477] Add variable management module
Posted by 陈卓宇 <mo...@gmail.com>.
The current controversy is not a core issue. Not adding `variable name`
would be a bit more international, because at the moment it looks like
`variable name` is
`variable code` is the Chinese meaning of `variable code`. I think we can
put more information under: `Description`.
Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
> The variable management module has three properties, including variable
> code, variable name, and description, For example, the variable code is
> "external.kafka.collect.brokers", and the variable name is "外部采集整合kafka".
> When we created the application, I forgot what the variable code was, but I
> knew it was called "采集整合", so I can search Enter the "采集整合" keywords in the
> box, and of course you can also search for kafka keywords.
>
> At the same time, I want to know more detailed information about this
> kafka, such as who the operator is, what is the data retention period, and
> how big is the cluster size, but I can't use the description to search, so
> I can only search or variable code or variable name, and the variable name
> is also for a simple display.
>
> Although external.kafka.collect.brokers already contains a lot of
> information, I think this is just a naming convention. When we discuss this
> variable, we will discuss it by using the "采集整合 kafka"
>
> We can discuss whether there is a problem with this design, what do you
> think?
>
> Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
>
> > Hi huajie:
> >
> > > If it is manually entered, it will always be treated as ordinary text.
> >
> > Users always copy some code from an old job to a new job.
> > I'm afraid it might not meet the user's expectations.
> >
> > Best,
> > fanrui
> >
> > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <be...@gmail.com> wrote:
> >
> > > hi 田向阳:
> > >
> > > The current discussion is about variable management, which is a
> > relatively
> > > independent function, No other modules involved, you can check the
> thread
> > > [1]
> > >
> > >
> > > [1] thread:
> > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > >
> > >
> > > Best,
> > > Huajie Wang
> > >
> > >
> > >
> > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > >
> > > > is this module include a resource management cente r where jars can
> be
> > > > chosen to provide to create a job
> > > >
> > > >
> > > >
> > > > ---- 回复的原邮件 ----
> > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > > | 日期 | 2022年10月08日 12:07 |
> > > > | 收件人 | dev@streampark.apache.org<de...@streampark.apache.org> |
> > > > | 抄送至 | |
> > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management module |
> > > > hi 阿穆:
> > > > Thank you for the clarification, My question is answered, but now I
> > have
> > > > one more concern,
> > > > I'm currently using {var} for variables, and I'm worried that one of
> > the
> > > > user's parameters includes "{" and "}", e.g :
> > > >
> > > > --password 'mypass{123}abc'
> > > >
> > > > So how do we parse parse this? How about ${var}
> > > >
> > > >
> > > >
> > > > Best,
> > > > Huajie Wang
> > > >
> > > >
> > > >
> > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> > > >
> > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > > >
> > > > > Thanks for your reply.
> > > > >
> > > > > I have integrated your replies. I really don't pay much attention
> to
> > > SQL,
> > > > > so I will do the following.
> > > > >
> > > > > 1.Each team has independent variables that do not affect each other
> > and
> > > > > are isolated from each other, but the same variable is unique under
> > the
> > > > > same team, so you need to specify the team when creating variables
> > > (users
> > > > > may belong to multiple teams, and the teams of the current user
> will
> > be
> > > > > displayed here for users to choose)
> > > > > 2.For Flink jar jobs, there are two ways to pass variables
> > > > > - By using placeholders, such as adding --kafka {mykafka} to the
> args
> > > of
> > > > > the application, 'mykafka' is the variable code of a variable
> > > > > - Search by variable code or variable name. After a variable is
> > > selected,
> > > > > the parameter transfer method is displayed
> > > > > 3.Flink sql will use variables as placeholders, E.g:
> > > > >
> > > > > CREATE TABLE KafkaTableA (
> > > > > `user_id` BIGINT,
> > > > > `item_id` BIGINT,
> > > > > `type` STRING
> > > > > ) WITH (
> > > > > 'connector' = 'kafka',
> > > > > 'topic' = 'user_behaviorA',
> > > > > 'properties.bootstrap.servers' =
> > '{kafka_clusterA}',
> > > > --
> > > > > 'kafka_ ClusterA' is the variable code
> > > > > 'properties.group.id' = 'testGroupA',
> > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > >
> > > > > CREATE TABLE KafkaTableB (
> > > > > `user_id` BIGINT,
> > > > > `item_id` BIGINT
> > > > > ) WITH (
> > > > > 'connector' = 'kafka',
> > > > > 'topic' = 'user_behaviorB',
> > > > > 'properties.bootstrap.servers' =
> > '{kafka_clusterB}',
> > > > --
> > > > > 'kafka_ ClusterB' is the variable code
> > > > > 'properties.group.id' = 'testGroupB',
> > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > >
> > > > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM
> > > > > KafkaTableB -- 'type' is the variable code
> > > > >
> > > > > 4.In the future, we can consider setting some system built-in
> > variables
> > > > >
> > > > > Best,
> > > > > ChunJin Mu
> > > > >
> > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <1996fanrui@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi 阿穆:
> > > > > >
> > > > > > It's a great proposal.
> > > > > >
> > > > > > For usage scenarios, I strongly recommend support for
> replacing
> > > > > variables
> > > > > > in sql. E.g:
> > > > > >
> > > > > > CREATE TABLE KafkaTable (
> > > > > > `user_id` BIGINT,
> > > > > > `item_id` BIGINT
> > > > > > ) WITH (
> > > > > > 'connector' = 'kafka',
> > > > > > 'topic' = 'user_behavior',
> > > > > > 'properties.bootstrap.servers' =
> > > > '{kafka_clusterA}',
> > > > > > 'properties.group.id' = 'testGroup',
> > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > 'format' = 'csv'
> > > > > > )
> > > > > >
> > > > > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > > > > >
> > > > > > For variable management, I have some question:
> > > > > >
> > > > > > 1. Is the parameter used within the team? If all teams use a
> > > common
> > > > > > parameter, it
> > > > > > needs to be defined for each team, right?
> > > > > > 2. Which users can modify variables? Can all users modify all
> > > > > variables of
> > > > > > their team?
> > > > > > After the variable is modified incorrectly, many jobs may be
> > > > > affected. So I
> > > > > > suggest that
> > > > > > only team admin or system admin have permission to modify, add
> > and
> > > > > delete
> > > > > > variables.
> > > > > >
> > > > > > Best
> > > > > > fanrui
> > > > >
> > > > >
> > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <benjobx@gmail.com>
> > > wrote:
> > > > >
> > > > > > hi 阿穆:
> > > > > >
> > > > > >
> > > > > > Thanks for starting a very useful discussion, I have a some
> > > > questions:
> > > > > >
> > > > > >
> > > > > > 1. The access boundary problem of variable management, is this
> > all
> > > > > > people in the platform can access these variables( no matter
> who
> > > > > > defines) or that only team members can be access these
> > variables.
> > > > > >
> > > > > > 2. There are two usage scenarios for variables that I know of,
> > one
> > > > is
> > > > > > in the entry parameter of main, and the other is in flinksql.
> > Are
> > > > both
> > > > > > supported, or only supported main entry ?
> > > > > >
> > > > > > about first question, my opinion:
> > > > > >
> > > > > > 1. The variable management is associated with the team. After
> > the
> > > > user
> > > > > > logs in to the system, he must select a team (a user can
> belong
> > to
> > > > one
> > > > > > or more teams), The variable belongs to a team
> > > > > >
> > > > > > 2. All resources between teams are isolated, which means that
> > > > > > variables are also isolated. In short, members of a team can
> use
> > > and
> > > > > > define these variables, which are invisible to other members,
> > and
> > > > > > other members cannot use these variables.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Welcome to your views
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Huajie Wang
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com> 于2022年10月7日周五 14:18写道：
> > > > >
> > > > > > > Is it possible to do this variable support in sql as well
> > > > > > >
> > > > > > > 阿穆 <305343894@qq.com.invalid> 于2022年10月7日周五 13:54写道：
> > > > > > >
> > > > > > > > Hi guys,
> > > > > > > >
> > > > > > > >
> > > > > > > > I want to start a discussion about the variable
> > > management
> > > > > module. Please
> > > > > > > > see this issue issue-1477.
> > > > > > > >
> > > > > > > >
> > > > > > > > Background:
> > > > > > > >
> > > > > > > >
> > > > > > > > In the actual production environment, Flink jobs are
> > > > > generally complex,
> > > > > > > > and usually require multiple external components.
> For
> > > > > example, Flink jobs
> > > > > > > > consume data from Kafka, then connect external
> > > components
> > > > > such as HBase
> > > > > > > or
> > > > > > > > Redis to obtain additional business information, and
> > > then
> > > > > write it to the
> > > > > > > > downstream external components. There are the
> > following
> > > > > problems:
> > > > > > > >
> > > > > > > >
> > > > > > > > 1. The connection information of external
> components,
> > > such
> > > > > as IP, port
> > > > > > > and
> > > > > > > > user password, needs to be configured in the
> > application
> > > > > args and
> > > > > > > > transferred to the Flink job, so that the connection
> > > > > information of
> > > > > > > > external components is distributed in multiple
> > > > > applications. Once the
> > > > > > > > connection information of external components
> changes,
> > > > many
> > > > > application
> > > > > > > > args parameters need to be modified, which will lead
> > to
> > > > > high operation
> > > > > > > and
> > > > > > > > maintenance costs.
> > > > > > > > 2. There are many people in the team developing
> > > real-time
> > > > > computing jobs.
> > > > > > > > There is no uniform specification for the connection
> > > > > information of the
> > > > > > > > external components passed to the job, resulting in
> > > > > different parameter
> > > > > > > > names of the same component. This is difficult to
> > count
> > > > > which external
> > > > > > > > components are dependent.
> > > > > > > > 3. In production practice, there are usually
> multiple
> > > sets
> > > > > of
> > > > > > > > environments, such as test environment and
> production
> > > > > environment. It is
> > > > > > > > not intuitive to judge whether a component belongs
> to
> > a
> > > > > test environment
> > > > > > > or
> > > > > > > > a production environment simply through IP and
> ports.
> > > Even
> > > > > if it can be
> > > > > > > > judged, there are some omissions. In this way, the
> > > > > connection information
> > > > > > > > online to the production environment may be external
> > > > > components of the
> > > > > > > test
> > > > > > > > environment, or components of the production
> > environment
> > > > are
> > > > > > > inadvertently
> > > > > > > > configured during testing, leading to production
> > > failures.
> > > > > > > >
> > > > > > > >
> > > > > > > > Proposal:
> > > > > > > >
> > > > > > > >
> > > > > > > > 1. Create a unified module to manage variables.
> Users
> > > can
> > > > > define their
> > > > > > > own
> > > > > > > > variables in this module. Variable information
> > includes
> > > > > variable code
> > > > > > > > (passed to Flink program as parameter name),
> variable
> > > > > value, variable
> > > > > > name,
> > > > > > > > variable description, and which applications depend
> on
> > > > > > > > 2. In the same Team module, the variable code or
> > > variable
> > > > > name is unique
> > > > > > > > - By using placeholders, such as adding -- kafka
> > > {mykafka}
> > > > > to the args of
> > > > > > > > the application, mykafka is the variable code of a
> > > > variable
> > > > > > > > - Search by variable code or variable name. After a
> > > > > variable is selected,
> > > > > > > > the parameter transfer method is displayed
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > If you have any suggestions, welcome to participate
> in
> > > the
> > > > > discussion on
> > > > > > > > the mail list or issue, I look forward to your
> > feedback.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best wishes
> > > > > > >
> > > >
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Chunjin Mu <ma...@gmail.com>.
hi guys.
When referencing variables in program args or Flink sql, in order to
enhance the user experience, follow-up search occurs when the user enters
"${", the user can support both variable code and description fuzzy search,
for example, if there is a variable code of a variable is
"internal.kafka.brokers", description is "内部kafka主机列表", when the user
enters "${kaf" and "${内部", this variable can be searched.
Do you think this design is feasible?
Best,
Chunjin Mu
Chunjin Mu <ma...@gmail.com> 于2022年10月15日周六 20:50写道：
> Hi fanru,
>
> Thank you very much for your hard review.
> You have considered it very comprehensively, I also saw your question in
> the PR, and I have already responded, please check this reply [1]
>
> [1]
> https://github.com/apache/incubator-streampark/pull/1831#discussion_r996293537
>
> Best,
> Chunjin Mu
>
> Rui Fan <19...@gmail.com> 于2022年10月15日周六 18:54写道：
>
>> Hi chunjin,
>>
>> Very sorry for slowing down your progress, I found some
>> small problems when I reviewed your code.
>>
>> Who can create the variable and update the variable?
>> System Admin and team_admin? Or every developer?
>>
>> As I understand, we didn't discuss this point, right? If I
>> miss any information please let me know, thanks a lot.
>>
>> Best,
>> fanrui
>>
>> On Sat, Oct 15, 2022 at 4:44 PM Chunjin Mu <ma...@gmail.com> wrote:
>>
>> > Hi fanru,
>> >
>> > > I have a little question. If I copy some sql from a old app, and the
>> sql
>> > includes some placeholders. These placeholders will take effect in the
>> > new app, right?
>> >
>> > yes, the copied placeholder will take effect if the variable exists.
>> >
>> > I have added all the information to proposal issue-1477[1]，include the
>> > questions you asked above， if there is more discussion later, I will add
>> > this to the proposal.
>> >
>> > [1] https://github.com/apache/incubator-streampark/issues/1477
>> >
>> >
>> > Rui Fan <19...@gmail.com> 于2022年10月15日周六 13:17写道：
>> >
>> > > Hi chunjin,
>> > >
>> > > Thanks for your detailed information.
>> > >
>> > > > When the focus is lost, it is judged whether there is a variable
>> > > placeholder, and if so, a verification button is displayed to prompt
>> the
>> > > user to perform verification. When the user clicks the verification
>> > button,
>> > > the replaced content of the placeholder is displayed, and only the
>> user
>> > > clicks the verification button to click the submit button
>> > >
>> > > I have a little question. If I copy some sql from a old app, and the
>> sql
>> > > includes some placeholders. These placeholders will take effect in the
>> > > new app, right?
>> > >
>> > > BTW, could you add all information to your proposal issue-1477[1]?
>> > > The mail list is for discussion only, and we should maintain the final
>> > > solution in the Proposal.
>> > >
>> > >
>> > > [1] https://github.com/apache/incubator-streampark/issues/1477
>> > >
>> > > Best
>> > > fanrui
>> > >
>> > > On Sat, Oct 15, 2022 at 11:56 AM Chunjin Mu <ma...@gmail.com>
>> > wrote:
>> > >
>> > > > Hi guys,
>> > > >
>> > > > Thanks for the discussion and great suggestions.
>> > > >
>> > > > Here I summarize the functions to be completed in version 1.2.4
>> > > >
>> > > > 1.The table structure of the variable contains the following main
>> > > contents
>> > > > - variable code is placeholder, For example the variable code is
>> > > > "collect.kafka.cluster", it will be referenced as
>> > > ${collect.kafka.cluster},
>> > > > it is unique within a team
>> > > > - variable name is removed, after discussion, it is not necessarily
>> > > > retained
>> > > > - variable value is the value of a specific variable, such as ip and
>> > port
>> > > > - description is a detailed introduction
>> > > > - team is the team the variable belongs to
>> > > >
>> > > > 2.Basic functions of variable management
>> > > > - Add and modify variables: need to check whether the variable code
>> is
>> > > > unique under the current team
>> > > > - Deleting a variable: It is necessary to check whether it has been
>> > > > referenced by the Application under the team to which the variable
>> > > belongs.
>> > > > If there is metadata management in the future, it is also necessary
>> to
>> > > > check whether it has been referenced by the metadata. If it is
>> > > referenced,
>> > > > it will prompt that the variable is referenced and cannot be
>> deleted.
>> > > >
>> > > > 3.References in Flink sql and program args
>> > > > - In the application, the variables are uniformly referenced in the
>> > form
>> > > of
>> > > > placeholders ${xxx}, such as in Flink sql, program args
>> > > > - For a better experience, it is necessary to provide the function
>> of
>> > > > following the search prompt when entering variable placeholders. For
>> > > > example, when entering "${", the variable list is displayed, and
>> when
>> > > > "${kafka" is entered, the display contains variables for kafka
>> strings,
>> > > in
>> > > > "variable code" or "description".
>> > > > - When the focus is lost, it is judged whether there is a variable
>> > > > placeholder, and if so, a verification button is displayed to prompt
>> > the
>> > > > user to perform verification. When the user clicks the verification
>> > > button,
>> > > > the replaced content of the placeholder is displayed, and only the
>> user
>> > > > clicks the verification button to click the submit button
>> > > > - If the user enters "okir23${j2Rts}pEt", this placeholder
>> "${j2Rts}"
>> > > will
>> > > > not be replaced if "j2Rts" is not a variable we defined
>> > > >
>> > > > 4.Future plan
>> > > > - Display the application or metadata list of a referenced variable
>> in
>> > > the
>> > > > variable basic management module
>> > > > - Improve user experience
>> > > >
>> > > > Best,
>> > > > Chunjin Mu
>> > > >
>> > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:28写道：
>> > > >
>> > > > > When we have dozens or dozens of variables, this phenomenon may be
>> > more
>> > > > > prominent
>> > > > >
>> > > > >
>> > > > > Best,
>> > > > > Chunjin Mu
>> > > > >
>> > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:26写道：
>> > > > >
>> > > > >> When a person asks which kafka should be used, I tell him in
>> Chinese
>> > > > that
>> > > > >> he should use "采集整合的kafka", in fact the variable code is
>> > > collect.kafka,
>> > > > but
>> > > > >> he is searching for gather keyword, it is difficult to just pass
>> > > > >> collect.kafka The intuitive understanding is to "采集整合的kafka",
>> > because
>> > > > >> the information contained in the variable code is still not that
>> > much.
>> > > > >>
>> > > > >> Huajie Wang <be...@gmail.com> 于2022年10月14日周五 22:17写道：
>> > > > >>
>> > > > >>> I just thought about it, there is really no need for this name.
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> Best,
>> > > > >>> Huajie Wang
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> Rui Fan <19...@gmail.com> 于2022年10月14日周五 21:54写道：
>> > > > >>>
>> > > > >>> > Hi guys,
>> > > > >>> >
>> > > > >>> > I understand chunjin's solution is this:
>> > > > >>> >
>> > > > >>> > - variable code is placeholder
>> > > > >>> > - variable name is an introduction
>> > > > >>> > - Description is a detailed introduction
>> > > > >>> >
>> > > > >>> > I think introduction and detail are redundant.
>> > > > >>> >
>> > > > >>> > For example: there are two kafka clusters, the Beijing kafka
>> > > > >>> > cluster and the Hangzhou kafka cluster.
>> > > > >>> >
>> > > > >>> > If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why
>> > > > >>> > does the introduction still need `北京 kafka 集群` and
>> > > > >>> > `杭州 kafka 集群`? placeholder is already clear.
>> > > > >>> >
>> > > > >>> > If dev may forget the placeholder, then dev may also
>> > > > >>> > forget the introduction.
>> > > > >>> >
>> > > > >>> > Of course, if the placeholder is kafka1 kafka2, an
>> > > > >>> > introduction is required. Why do we need an introduction
>> > > > >>> > at this point, because the placeholder name is unreasonable.
>> > > > >>> >
>> > > > >>> > Just like variable naming in Java code, there is no need to
>> > > > >>> > introduce an introduction with a clear name. If the meaning of
>> > > > >>> > the variable is complicated, we also need to write a detailed
>> > > > >>> > description.
>> > > > >>> >
>> > > > >>> > Finally, I want to say: less is more. If we want to make
>> > > > >>> > something complicated, please give everyone a strong reason.
>> > > > >>> >
>> > > > >>> > Best
>> > > > >>> > fanrui
>> > > > >>> >
>> > > > >>> > On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <
>> benjobx@gmail.com>
>> > > > wrote:
>> > > > >>> >
>> > > > >>> > > Hi guys,
>> > > > >>> > >
>> > > > >>> > > I agree with Chunjin, I think variable name is useful,
>> Chunjin
>> > > also
>> > > > >>> > > explained the reason.
>> > > > >>> > >
>> > > > >>> > >
>> > > > >>> > > Best,
>> > > > >>> > > Huajie Wang
>> > > > >>> > >
>> > > > >>> > >
>> > > > >>> > >
>> > > > >>> > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 20:22写道：
>> > > > >>> > >
>> > > > >>> > > > Hi, Rui Fan
>> > > > >>> > > > I'll start a discussion later.
>> > > > >>> > > >
>> > > > >>> > > > Best
>> > > > >>> > > > Chunjin Mu
>> > > > >>> > > >
>> > > > >>> > > > Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
>> > > > >>> > > >
>> > > > >>> > > > > Hi guys,
>> > > > >>> > > > >
>> > > > >>> > > > > I agree with monster. And I suggest use variable name,
>> > > variable
>> > > > >>> value
>> > > > >>> > > and
>> > > > >>> > > > > description.
>> > > > >>> > > > >
>> > > > >>> > > > > By the way, there are some problems still need to
>> confirm.
>> > > > >>> What’s the
>> > > > >>> > > > > placeholder? And how to use them when edit sql or
>> > datastream
>> > > > job?
>> > > > >>> > It’s
>> > > > >>> > > > > better to discuss these asap.
>> > > > >>> > > > >
>> > > > >>> > > > > Best
>> > > > >>> > > > > fanrui
>> > > > >>> > > > >
>> > > > >>> > > > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <
>> > > macksonmu0@gmail.com
>> > > > >
>> > > > >>> > wrote:
>> > > > >>> > > > >
>> > > > >>> > > > > > hi, huajie, fan rui, monster
>> > > > >>> > > > > >
>> > > > >>> > > > > > Thank you for your suggestions.
>> > > > >>> > > > > >
>> > > > >>> > > > > > Now please Huajie to confirm how to do it. If remove
>> the
>> > > > >>> variable
>> > > > >>> > > > name, I
>> > > > >>> > > > > > can also accept it here. I will do small secondary
>> > > > development
>> > > > >>> on
>> > > > >>> > our
>> > > > >>> > > > > > production system based on the community to meet
>> internal
>> > > > >>> needs.
>> > > > >>> > > > > >
>> > > > >>> > > > > > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五
>> > 15:58写道：
>> > > > >>> > > > > >
>> > > > >>> > > > > > > "外部采集整合kafka" is the Chinese description of
>> variable,
>> > so
>> > > it
>> > > > >>> makes
>> > > > >>> > > > sense
>> > > > >>> > > > > > to
>> > > > >>> > > > > > > put it under `Description`.
>> > > > >>> > > > > > >
>> > > > >>> > > > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五
>> > > 12:01写道：
>> > > > >>> > > > > > >
>> > > > >>> > > > > > > > The variable management module has three
>> properties,
>> > > > >>> including
>> > > > >>> > > > > variable
>> > > > >>> > > > > > > > code, variable name, and description, For example,
>> > the
>> > > > >>> variable
>> > > > >>> > > > code
>> > > > >>> > > > > is
>> > > > >>> > > > > > > > "external.kafka.collect.brokers", and the variable
>> > name
>> > > > is
>> > > > >>> > > > > > "外部采集整合kafka".
>> > > > >>> > > > > > > > When we created the application, I forgot what the
>> > > > variable
>> > > > >>> > code
>> > > > >>> > > > was,
>> > > > >>> > > > > > > but I
>> > > > >>> > > > > > > > knew it was called "采集整合", so I can search Enter
>> the
>> > > > "采集整合"
>> > > > >>> > > > keywords
>> > > > >>> > > > > in
>> > > > >>> > > > > > > the
>> > > > >>> > > > > > > > box, and of course you can also search for kafka
>> > > > keywords.
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > At the same time, I want to know more detailed
>> > > > information
>> > > > >>> > about
>> > > > >>> > > > this
>> > > > >>> > > > > > > > kafka, such as who the operator is, what is the
>> data
>> > > > >>> retention
>> > > > >>> > > > > period,
>> > > > >>> > > > > > > and
>> > > > >>> > > > > > > > how big is the cluster size, but I can't use the
>> > > > >>> description to
>> > > > >>> > > > > search,
>> > > > >>> > > > > > > so
>> > > > >>> > > > > > > > I can only search or variable code or variable
>> name,
>> > > and
>> > > > >>> the
>> > > > >>> > > > variable
>> > > > >>> > > > > > > name
>> > > > >>> > > > > > > > is also for a simple display.
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > Although external.kafka.collect.brokers already
>> > > contains
>> > > > a
>> > > > >>> lot
>> > > > >>> > of
>> > > > >>> > > > > > > > information, I think this is just a naming
>> > convention.
>> > > > >>> When we
>> > > > >>> > > > > discuss
>> > > > >>> > > > > > > this
>> > > > >>> > > > > > > > variable, we will discuss it by using the "采集整合
>> > kafka"
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > We can discuss whether there is a problem with
>> this
>> > > > design,
>> > > > >>> > what
>> > > > >>> > > do
>> > > > >>> > > > > you
>> > > > >>> > > > > > > > think?
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一
>> > 11:45写道：
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > > Hi huajie:
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > > > > If it is manually entered, it will always be
>> > > treated
>> > > > as
>> > > > >>> > > > ordinary
>> > > > >>> > > > > > > text.
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > > > Users always copy some code from an old job to a
>> > new
>> > > > job.
>> > > > >>> > > > > > > > > I'm afraid it might not meet the user's
>> > expectations.
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > > > Best,
>> > > > >>> > > > > > > > > fanrui
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <
>> > > > >>> > benjobx@gmail.com>
>> > > > >>> > > > > > wrote:
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > > > > hi 田向阳:
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > > The current discussion is about variable
>> > > management,
>> > > > >>> which
>> > > > >>> > > is a
>> > > > >>> > > > > > > > > relatively
>> > > > >>> > > > > > > > > > independent function, No other modules
>> involved,
>> > > you
>> > > > >>> can
>> > > > >>> > > check
>> > > > >>> > > > > the
>> > > > >>> > > > > > > > thread
>> > > > >>> > > > > > > > > > [1]
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > > [1] thread:
>> > > > >>> > > > > > > > > >
>> > > > >>> > > >
>> > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > > Best,
>> > > > >>> > > > > > > > > > Huajie Wang
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六
>> 13:13写道：
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > > > > is this module include a resource management
>> > > cente
>> > > > r
>> > > > >>> > where
>> > > > >>> > > > jars
>> > > > >>> > > > > > can
>> > > > >>> > > > > > > > be
>> > > > >>> > > > > > > > > > > chosen to provide to create a job
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > ---- 回复的原邮件 ----
>> > > > >>> > > > > > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
>> > > > >>> > > > > > > > > > > | 日期 | 2022年10月08日 12:07 |
>> > > > >>> > > > > > > > > > > | 收件人 | dev@streampark.apache.org<
>> > > > >>> > > dev@streampark.apache.org>
>> > > > >>> > > > |
>> > > > >>> > > > > > > > > > > | 抄送至 | |
>> > > > >>> > > > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add
>> variable
>> > > > >>> management
>> > > > >>> > > > > module |
>> > > > >>> > > > > > > > > > > hi 阿穆:
>> > > > >>> > > > > > > > > > > Thank you for the clarification, My
>> question is
>> > > > >>> answered,
>> > > > >>> > > but
>> > > > >>> > > > > > now I
>> > > > >>> > > > > > > > > have
>> > > > >>> > > > > > > > > > > one more concern,
>> > > > >>> > > > > > > > > > > I'm currently using {var} for variables, and
>> > I'm
>> > > > >>> worried
>> > > > >>> > > that
>> > > > >>> > > > > one
>> > > > >>> > > > > > > of
>> > > > >>> > > > > > > > > the
>> > > > >>> > > > > > > > > > > user's parameters includes "{" and "}", e.g
>> :
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > --password 'mypass{123}abc'
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > So how do we parse parse this? How about
>> > ${var}
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > Best,
>> > > > >>> > > > > > > > > > > Huajie Wang
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六
>> > > > 11:36写道：
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > Thanks for your reply.
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > I have integrated your replies. I really
>> > don't
>> > > > pay
>> > > > >>> much
>> > > > >>> > > > > > attention
>> > > > >>> > > > > > > > to
>> > > > >>> > > > > > > > > > SQL,
>> > > > >>> > > > > > > > > > > > so I will do the following.
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > 1.Each team has independent variables
>> that do
>> > > not
>> > > > >>> > affect
>> > > > >>> > > > each
>> > > > >>> > > > > > > other
>> > > > >>> > > > > > > > > and
>> > > > >>> > > > > > > > > > > > are isolated from each other, but the same
>> > > > >>> variable is
>> > > > >>> > > > unique
>> > > > >>> > > > > > > under
>> > > > >>> > > > > > > > > the
>> > > > >>> > > > > > > > > > > > same team, so you need to specify the team
>> > when
>> > > > >>> > creating
>> > > > >>> > > > > > > variables
>> > > > >>> > > > > > > > > > (users
>> > > > >>> > > > > > > > > > > > may belong to multiple teams, and the
>> teams
>> > of
>> > > > the
>> > > > >>> > > current
>> > > > >>> > > > > user
>> > > > >>> > > > > > > > will
>> > > > >>> > > > > > > > > be
>> > > > >>> > > > > > > > > > > > displayed here for users to choose)
>> > > > >>> > > > > > > > > > > > 2.For Flink jar jobs, there are two ways
>> to
>> > > pass
>> > > > >>> > > variables
>> > > > >>> > > > > > > > > > > > - By using placeholders, such as adding
>> > --kafka
>> > > > >>> > {mykafka}
>> > > > >>> > > > to
>> > > > >>> > > > > > the
>> > > > >>> > > > > > > > args
>> > > > >>> > > > > > > > > > of
>> > > > >>> > > > > > > > > > > > the application, 'mykafka' is the variable
>> > code
>> > > > of
>> > > > >>> a
>> > > > >>> > > > variable
>> > > > >>> > > > > > > > > > > > - Search by variable code or variable
>> name.
>> > > > After a
>> > > > >>> > > > variable
>> > > > >>> > > > > is
>> > > > >>> > > > > > > > > > selected,
>> > > > >>> > > > > > > > > > > > the parameter transfer method is displayed
>> > > > >>> > > > > > > > > > > > 3.Flink sql will use variables as
>> > placeholders,
>> > > > >>> E.g:
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > CREATE TABLE KafkaTableA (
>> > > > >>> > > > > > > > > > > > `user_id` BIGINT,
>> > > > >>> > > > > > > > > > > > `item_id` BIGINT,
>> > > > >>> > > > > > > > > > > > `type` STRING
>> > > > >>> > > > > > > > > > > > ) WITH (
>> > > > >>> > > > > > > > > > > > 'connector' = 'kafka',
>> > > > >>> > > > > > > > > > > > 'topic' = 'user_behaviorA',
>> > > > >>> > > > > > > > > > > >
>> 'properties.bootstrap.servers' =
>> > > > >>> > > > > > > > > '{kafka_clusterA}',
>> > > > >>> > > > > > > > > > > --
>> > > > >>> > > > > > > > > > > > 'kafka_ ClusterA' is the variable code
>> > > > >>> > > > > > > > > > > > 'properties.group.id' =
>> > > > 'testGroupA',
>> > > > >>> > > > > > > > > > > > 'scan.startup.mode' =
>> > > > >>> 'earliest-offset',
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > CREATE TABLE KafkaTableB (
>> > > > >>> > > > > > > > > > > > `user_id` BIGINT,
>> > > > >>> > > > > > > > > > > > `item_id` BIGINT
>> > > > >>> > > > > > > > > > > > ) WITH (
>> > > > >>> > > > > > > > > > > > 'connector' = 'kafka',
>> > > > >>> > > > > > > > > > > > 'topic' = 'user_behaviorB',
>> > > > >>> > > > > > > > > > > >
>> 'properties.bootstrap.servers' =
>> > > > >>> > > > > > > > > '{kafka_clusterB}',
>> > > > >>> > > > > > > > > > > --
>> > > > >>> > > > > > > > > > > > 'kafka_ ClusterB' is the variable code
>> > > > >>> > > > > > > > > > > > 'properties.group.id' =
>> > > > 'testGroupB',
>> > > > >>> > > > > > > > > > > > 'scan.startup.mode' =
>> > > > >>> 'earliest-offset',
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id,
>> > > item_id,
>> > > > >>> > '{type}'
>> > > > >>> > > > > FROM
>> > > > >>> > > > > > > > > > > > KafkaTableB -- 'type' is the
>> > > variable
>> > > > >>> code
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > 4.In the future, we can consider setting
>> some
>> > > > >>> system
>> > > > >>> > > > built-in
>> > > > >>> > > > > > > > > variables
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > Best,
>> > > > >>> > > > > > > > > > > > ChunJin Mu
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
>> > > > >>> > > > 1996fanrui@gmail.com
>> > > > >>> > > > > > >
>> > > > >>> > > > > > > > > > wrote:
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Hi 阿穆:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > It's a great proposal.
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > For usage scenarios, I strongly
>> > recommend
>> > > > >>> support
>> > > > >>> > > for
>> > > > >>> > > > > > > > replacing
>> > > > >>> > > > > > > > > > > > variables
>> > > > >>> > > > > > > > > > > > > in sql. E.g:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > CREATE TABLE KafkaTable (
>> > > > >>> > > > > > > > > > > > > `user_id` BIGINT,
>> > > > >>> > > > > > > > > > > > > `item_id` BIGINT
>> > > > >>> > > > > > > > > > > > > ) WITH (
>> > > > >>> > > > > > > > > > > > > 'connector' =
>> 'kafka',
>> > > > >>> > > > > > > > > > > > > 'topic' =
>> > > 'user_behavior',
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> 'properties.bootstrap.servers' =
>> > > > >>> > > > > > > > > > > '{kafka_clusterA}',
>> > > > >>> > > > > > > > > > > > > '
>> properties.group.id'
>> > =
>> > > > >>> > > > 'testGroup',
>> > > > >>> > > > > > > > > > > > >
>> 'scan.startup.mode' =
>> > > > >>> > > > > 'earliest-offset',
>> > > > >>> > > > > > > > > > > > > 'format' = 'csv'
>> > > > >>> > > > > > > > > > > > > )
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Note: 'properties.bootstrap.servers'
>> =
>> > > > >>> > > > > '{kafka_clusterA}',
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > For variable management, I have some
>> > > > question:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 1. Is the parameter used within the
>> > team?
>> > > If
>> > > > >>> all
>> > > > >>> > > teams
>> > > > >>> > > > > > use a
>> > > > >>> > > > > > > > > > common
>> > > > >>> > > > > > > > > > > > > parameter, it
>> > > > >>> > > > > > > > > > > > > needs to be defined for each team,
>> > right?
>> > > > >>> > > > > > > > > > > > > 2. Which users can modify variables?
>> Can
>> > > all
>> > > > >>> users
>> > > > >>> > > > > modify
>> > > > >>> > > > > > > all
>> > > > >>> > > > > > > > > > > > variables of
>> > > > >>> > > > > > > > > > > > > their team?
>> > > > >>> > > > > > > > > > > > > After the variable is modified
>> > > incorrectly,
>> > > > >>> many
>> > > > >>> > > jobs
>> > > > >>> > > > > may
>> > > > >>> > > > > > be
>> > > > >>> > > > > > > > > > > > affected. So I
>> > > > >>> > > > > > > > > > > > > suggest that
>> > > > >>> > > > > > > > > > > > > only team admin or system admin have
>> > > > >>> permission to
>> > > > >>> > > > > modify,
>> > > > >>> > > > > > > add
>> > > > >>> > > > > > > > > and
>> > > > >>> > > > > > > > > > > > delete
>> > > > >>> > > > > > > > > > > > > variables.
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Best
>> > > > >>> > > > > > > > > > > > > fanrui
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie
>> Wang <
>> > > > >>> > > > > benjobx@gmail.com
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > > > > > wrote:
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > hi 阿穆:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Thanks for starting a very useful
>> > > > discussion,
>> > > > >>> I
>> > > > >>> > > have a
>> > > > >>> > > > > > some
>> > > > >>> > > > > > > > > > > questions:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 1. The access boundary problem of
>> > variable
>> > > > >>> > > management,
>> > > > >>> > > > > is
>> > > > >>> > > > > > > this
>> > > > >>> > > > > > > > > all
>> > > > >>> > > > > > > > > > > > > people in the platform can access
>> these
>> > > > >>> variables(
>> > > > >>> > > no
>> > > > >>> > > > > > matter
>> > > > >>> > > > > > > > who
>> > > > >>> > > > > > > > > > > > > defines) or that only team members
>> can
>> > be
>> > > > >>> access
>> > > > >>> > > these
>> > > > >>> > > > > > > > > variables.
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 2. There are two usage scenarios for
>> > > > variables
>> > > > >>> > that
>> > > > >>> > > I
>> > > > >>> > > > > know
>> > > > >>> > > > > > > of,
>> > > > >>> > > > > > > > > one
>> > > > >>> > > > > > > > > > > is
>> > > > >>> > > > > > > > > > > > > in the entry parameter of main, and
>> the
>> > > > other
>> > > > >>> is
>> > > > >>> > in
>> > > > >>> > > > > > > flinksql.
>> > > > >>> > > > > > > > > Are
>> > > > >>> > > > > > > > > > > both
>> > > > >>> > > > > > > > > > > > > supported, or only supported main
>> entry
>> > ?
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > about first question, my opinion:
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 1. The variable management is
>> associated
>> > > > with
>> > > > >>> the
>> > > > >>> > > > team.
>> > > > >>> > > > > > > After
>> > > > >>> > > > > > > > > the
>> > > > >>> > > > > > > > > > > user
>> > > > >>> > > > > > > > > > > > > logs in to the system, he must
>> select a
>> > > team
>> > > > >>> (a
>> > > > >>> > user
>> > > > >>> > > > can
>> > > > >>> > > > > > > > belong
>> > > > >>> > > > > > > > > to
>> > > > >>> > > > > > > > > > > one
>> > > > >>> > > > > > > > > > > > > or more teams), The variable belongs
>> to
>> > a
>> > > > team
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 2. All resources between teams are
>> > > isolated,
>> > > > >>> which
>> > > > >>> > > > means
>> > > > >>> > > > > > > that
>> > > > >>> > > > > > > > > > > > > variables are also isolated. In
>> short,
>> > > > >>> members of
>> > > > >>> > a
>> > > > >>> > > > team
>> > > > >>> > > > > > can
>> > > > >>> > > > > > > > use
>> > > > >>> > > > > > > > > > and
>> > > > >>> > > > > > > > > > > > > define these variables, which are
>> > > invisible
>> > > > to
>> > > > >>> > other
>> > > > >>> > > > > > > members,
>> > > > >>> > > > > > > > > and
>> > > > >>> > > > > > > > > > > > > other members cannot use these
>> > variables.
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Welcome to your views
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > Best,
>> > > > >>> > > > > > > > > > > > > Huajie Wang
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com
>> >
>> > > > >>> > 于2022年10月7日周五
>> > > > >>> > > > > > > 14:18写道：
>> > > > >>> > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > Is it possible to do this
>> variable
>> > > > >>> support in
>> > > > >>> > > sql
>> > > > >>> > > > > as
>> > > > >>> > > > > > > well
>> > > > >>> > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > 阿穆 <305343894@qq.com.invalid
>> >
>> > > > >>> > 于2022年10月7日周五
>> > > > >>> > > > > > > 13:54写道：
>> > > > >>> > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > Hi guys,
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > I want to start a
>> discussion
>> > > about
>> > > > >>> the
>> > > > >>> > > > > variable
>> > > > >>> > > > > > > > > > management
>> > > > >>> > > > > > > > > > > > module. Please
>> > > > >>> > > > > > > > > > > > > > > see this issue issue-1477.
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > Background:
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > In the actual production
>> > > > >>> environment,
>> > > > >>> > > Flink
>> > > > >>> > > > > jobs
>> > > > >>> > > > > > > are
>> > > > >>> > > > > > > > > > > > generally complex,
>> > > > >>> > > > > > > > > > > > > > > and usually require
>> multiple
>> > > > >>> external
>> > > > >>> > > > > > components.
>> > > > >>> > > > > > > > For
>> > > > >>> > > > > > > > > > > > example, Flink jobs
>> > > > >>> > > > > > > > > > > > > > > consume data from Kafka,
>> then
>> > > > >>> connect
>> > > > >>> > > > external
>> > > > >>> > > > > > > > > > components
>> > > > >>> > > > > > > > > > > > such as HBase
>> > > > >>> > > > > > > > > > > > > > or
>> > > > >>> > > > > > > > > > > > > > > Redis to obtain additional
>> > > > business
>> > > > >>> > > > > information,
>> > > > >>> > > > > > > and
>> > > > >>> > > > > > > > > > then
>> > > > >>> > > > > > > > > > > > write it to the
>> > > > >>> > > > > > > > > > > > > > > downstream external
>> > components.
>> > > > >>> There
>> > > > >>> > are
>> > > > >>> > > > the
>> > > > >>> > > > > > > > > following
>> > > > >>> > > > > > > > > > > > problems:
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > 1. The connection
>> information
>> > of
>> > > > >>> > external
>> > > > >>> > > > > > > > components,
>> > > > >>> > > > > > > > > > such
>> > > > >>> > > > > > > > > > > > as IP, port
>> > > > >>> > > > > > > > > > > > > > and
>> > > > >>> > > > > > > > > > > > > > > user password, needs to be
>> > > > >>> configured in
>> > > > >>> > > the
>> > > > >>> > > > > > > > > application
>> > > > >>> > > > > > > > > > > > args and
>> > > > >>> > > > > > > > > > > > > > > transferred to the Flink
>> job,
>> > so
>> > > > >>> that
>> > > > >>> > the
>> > > > >>> > > > > > > connection
>> > > > >>> > > > > > > > > > > > information of
>> > > > >>> > > > > > > > > > > > > > > external components is
>> > > distributed
>> > > > >>> in
>> > > > >>> > > > multiple
>> > > > >>> > > > > > > > > > > > applications. Once the
>> > > > >>> > > > > > > > > > > > > > > connection information of
>> > > external
>> > > > >>> > > > components
>> > > > >>> > > > > > > > changes,
>> > > > >>> > > > > > > > > > > many
>> > > > >>> > > > > > > > > > > > application
>> > > > >>> > > > > > > > > > > > > > > args parameters need to be
>> > > > modified,
>> > > > >>> > which
>> > > > >>> > > > > will
>> > > > >>> > > > > > > lead
>> > > > >>> > > > > > > > > to
>> > > > >>> > > > > > > > > > > > high operation
>> > > > >>> > > > > > > > > > > > > > and
>> > > > >>> > > > > > > > > > > > > > > maintenance costs.
>> > > > >>> > > > > > > > > > > > > > > 2. There are many people in
>> > the
>> > > > team
>> > > > >>> > > > > developing
>> > > > >>> > > > > > > > > > real-time
>> > > > >>> > > > > > > > > > > > computing jobs.
>> > > > >>> > > > > > > > > > > > > > > There is no uniform
>> > > specification
>> > > > >>> for
>> > > > >>> > the
>> > > > >>> > > > > > > connection
>> > > > >>> > > > > > > > > > > > information of the
>> > > > >>> > > > > > > > > > > > > > > external components passed
>> to
>> > > the
>> > > > >>> job,
>> > > > >>> > > > > resulting
>> > > > >>> > > > > > > in
>> > > > >>> > > > > > > > > > > > different parameter
>> > > > >>> > > > > > > > > > > > > > > names of the same
>> component.
>> > > This
>> > > > is
>> > > > >>> > > > difficult
>> > > > >>> > > > > > to
>> > > > >>> > > > > > > > > count
>> > > > >>> > > > > > > > > > > > which external
>> > > > >>> > > > > > > > > > > > > > > components are dependent.
>> > > > >>> > > > > > > > > > > > > > > 3. In production practice,
>> > there
>> > > > are
>> > > > >>> > > usually
>> > > > >>> > > > > > > > multiple
>> > > > >>> > > > > > > > > > sets
>> > > > >>> > > > > > > > > > > > of
>> > > > >>> > > > > > > > > > > > > > > environments, such as test
>> > > > >>> environment
>> > > > >>> > and
>> > > > >>> > > > > > > > production
>> > > > >>> > > > > > > > > > > > environment. It is
>> > > > >>> > > > > > > > > > > > > > > not intuitive to judge
>> > whether a
>> > > > >>> > component
>> > > > >>> > > > > > belongs
>> > > > >>> > > > > > > > to
>> > > > >>> > > > > > > > > a
>> > > > >>> > > > > > > > > > > > test environment
>> > > > >>> > > > > > > > > > > > > > or
>> > > > >>> > > > > > > > > > > > > > > a production environment
>> > simply
>> > > > >>> through
>> > > > >>> > IP
>> > > > >>> > > > and
>> > > > >>> > > > > > > > ports.
>> > > > >>> > > > > > > > > > Even
>> > > > >>> > > > > > > > > > > > if it can be
>> > > > >>> > > > > > > > > > > > > > > judged, there are some
>> > > omissions.
>> > > > In
>> > > > >>> > this
>> > > > >>> > > > way,
>> > > > >>> > > > > > the
>> > > > >>> > > > > > > > > > > > connection information
>> > > > >>> > > > > > > > > > > > > > > online to the production
>> > > > >>> environment may
>> > > > >>> > > be
>> > > > >>> > > > > > > external
>> > > > >>> > > > > > > > > > > > components of the
>> > > > >>> > > > > > > > > > > > > > test
>> > > > >>> > > > > > > > > > > > > > > environment, or components
>> of
>> > > the
>> > > > >>> > > production
>> > > > >>> > > > > > > > > environment
>> > > > >>> > > > > > > > > > > are
>> > > > >>> > > > > > > > > > > > > > inadvertently
>> > > > >>> > > > > > > > > > > > > > > configured during testing,
>> > > leading
>> > > > >>> to
>> > > > >>> > > > > production
>> > > > >>> > > > > > > > > > failures.
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > Proposal:
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > 1. Create a unified module
>> to
>> > > > manage
>> > > > >>> > > > > variables.
>> > > > >>> > > > > > > > Users
>> > > > >>> > > > > > > > > > can
>> > > > >>> > > > > > > > > > > > define their
>> > > > >>> > > > > > > > > > > > > > own
>> > > > >>> > > > > > > > > > > > > > > variables in this module.
>> > > Variable
>> > > > >>> > > > information
>> > > > >>> > > > > > > > > includes
>> > > > >>> > > > > > > > > > > > variable code
>> > > > >>> > > > > > > > > > > > > > > (passed to Flink program as
>> > > > >>> parameter
>> > > > >>> > > name),
>> > > > >>> > > > > > > > variable
>> > > > >>> > > > > > > > > > > > value, variable
>> > > > >>> > > > > > > > > > > > > name,
>> > > > >>> > > > > > > > > > > > > > > variable description, and
>> > which
>> > > > >>> > > applications
>> > > > >>> > > > > > > depend
>> > > > >>> > > > > > > > on
>> > > > >>> > > > > > > > > > > > > > > 2. In the same Team module,
>> > the
>> > > > >>> variable
>> > > > >>> > > > code
>> > > > >>> > > > > or
>> > > > >>> > > > > > > > > > variable
>> > > > >>> > > > > > > > > > > > name is unique
>> > > > >>> > > > > > > > > > > > > > > - By using placeholders,
>> such
>> > as
>> > > > >>> adding
>> > > > >>> > --
>> > > > >>> > > > > kafka
>> > > > >>> > > > > > > > > > {mykafka}
>> > > > >>> > > > > > > > > > > > to the args of
>> > > > >>> > > > > > > > > > > > > > > the application, mykafka is
>> > the
>> > > > >>> variable
>> > > > >>> > > > code
>> > > > >>> > > > > > of a
>> > > > >>> > > > > > > > > > > variable
>> > > > >>> > > > > > > > > > > > > > > - Search by variable code
>> or
>> > > > >>> variable
>> > > > >>> > > name.
>> > > > >>> > > > > > After
>> > > > >>> > > > > > > a
>> > > > >>> > > > > > > > > > > > variable is selected,
>> > > > >>> > > > > > > > > > > > > > > the parameter transfer
>> method
>> > is
>> > > > >>> > displayed
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > If you have any
>> suggestions,
>> > > > >>> welcome to
>> > > > >>> > > > > > > participate
>> > > > >>> > > > > > > > in
>> > > > >>> > > > > > > > > > the
>> > > > >>> > > > > > > > > > > > discussion on
>> > > > >>> > > > > > > > > > > > > > > the mail list or issue, I
>> look
>> > > > >>> forward
>> > > > >>> > to
>> > > > >>> > > > your
>> > > > >>> > > > > > > > > feedback.
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > > > > > > Best wishes
>> > > > >>> > > > > > > > > > > > > >
>> > > > >>> > > > > > > > > > >
>> > > > >>> > > > > > > > > >
>> > > > >>> > > > > > > > >
>> > > > >>> > > > > > > >
>> > > > >>> > > > > > >
>> > > > >>> > > > > >
>> > > > >>> > > > >
>> > > > >>> > > >
>> > > > >>> > >
>> > > > >>> >
>> > > > >>>
>> > > > >>
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Chunjin Mu <ma...@gmail.com>.
Hi fanru,
Thank you very much for your hard review.
You have considered it very comprehensively, I also saw your question in
the PR, and I have already responded, please check this reply [1]
[1]
https://github.com/apache/incubator-streampark/pull/1831#discussion_r996293537
Best,
Chunjin Mu
Rui Fan <19...@gmail.com> 于2022年10月15日周六 18:54写道：
> Hi chunjin,
>
> Very sorry for slowing down your progress, I found some
> small problems when I reviewed your code.
>
> Who can create the variable and update the variable?
> System Admin and team_admin? Or every developer?
>
> As I understand, we didn't discuss this point, right? If I
> miss any information please let me know, thanks a lot.
>
> Best,
> fanrui
>
> On Sat, Oct 15, 2022 at 4:44 PM Chunjin Mu <ma...@gmail.com> wrote:
>
> > Hi fanru,
> >
> > > I have a little question. If I copy some sql from a old app, and the
> sql
> > includes some placeholders. These placeholders will take effect in the
> > new app, right?
> >
> > yes, the copied placeholder will take effect if the variable exists.
> >
> > I have added all the information to proposal issue-1477[1]，include the
> > questions you asked above， if there is more discussion later, I will add
> > this to the proposal.
> >
> > [1] https://github.com/apache/incubator-streampark/issues/1477
> >
> >
> > Rui Fan <19...@gmail.com> 于2022年10月15日周六 13:17写道：
> >
> > > Hi chunjin,
> > >
> > > Thanks for your detailed information.
> > >
> > > > When the focus is lost, it is judged whether there is a variable
> > > placeholder, and if so, a verification button is displayed to prompt
> the
> > > user to perform verification. When the user clicks the verification
> > button,
> > > the replaced content of the placeholder is displayed, and only the user
> > > clicks the verification button to click the submit button
> > >
> > > I have a little question. If I copy some sql from a old app, and the
> sql
> > > includes some placeholders. These placeholders will take effect in the
> > > new app, right?
> > >
> > > BTW, could you add all information to your proposal issue-1477[1]?
> > > The mail list is for discussion only, and we should maintain the final
> > > solution in the Proposal.
> > >
> > >
> > > [1] https://github.com/apache/incubator-streampark/issues/1477
> > >
> > > Best
> > > fanrui
> > >
> > > On Sat, Oct 15, 2022 at 11:56 AM Chunjin Mu <ma...@gmail.com>
> > wrote:
> > >
> > > > Hi guys,
> > > >
> > > > Thanks for the discussion and great suggestions.
> > > >
> > > > Here I summarize the functions to be completed in version 1.2.4
> > > >
> > > > 1.The table structure of the variable contains the following main
> > > contents
> > > > - variable code is placeholder, For example the variable code is
> > > > "collect.kafka.cluster", it will be referenced as
> > > ${collect.kafka.cluster},
> > > > it is unique within a team
> > > > - variable name is removed, after discussion, it is not necessarily
> > > > retained
> > > > - variable value is the value of a specific variable, such as ip and
> > port
> > > > - description is a detailed introduction
> > > > - team is the team the variable belongs to
> > > >
> > > > 2.Basic functions of variable management
> > > > - Add and modify variables: need to check whether the variable code
> is
> > > > unique under the current team
> > > > - Deleting a variable: It is necessary to check whether it has been
> > > > referenced by the Application under the team to which the variable
> > > belongs.
> > > > If there is metadata management in the future, it is also necessary
> to
> > > > check whether it has been referenced by the metadata. If it is
> > > referenced,
> > > > it will prompt that the variable is referenced and cannot be deleted.
> > > >
> > > > 3.References in Flink sql and program args
> > > > - In the application, the variables are uniformly referenced in the
> > form
> > > of
> > > > placeholders ${xxx}, such as in Flink sql, program args
> > > > - For a better experience, it is necessary to provide the function of
> > > > following the search prompt when entering variable placeholders. For
> > > > example, when entering "${", the variable list is displayed, and when
> > > > "${kafka" is entered, the display contains variables for kafka
> strings,
> > > in
> > > > "variable code" or "description".
> > > > - When the focus is lost, it is judged whether there is a variable
> > > > placeholder, and if so, a verification button is displayed to prompt
> > the
> > > > user to perform verification. When the user clicks the verification
> > > button,
> > > > the replaced content of the placeholder is displayed, and only the
> user
> > > > clicks the verification button to click the submit button
> > > > - If the user enters "okir23${j2Rts}pEt", this placeholder "${j2Rts}"
> > > will
> > > > not be replaced if "j2Rts" is not a variable we defined
> > > >
> > > > 4.Future plan
> > > > - Display the application or metadata list of a referenced variable
> in
> > > the
> > > > variable basic management module
> > > > - Improve user experience
> > > >
> > > > Best,
> > > > Chunjin Mu
> > > >
> > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:28写道：
> > > >
> > > > > When we have dozens or dozens of variables, this phenomenon may be
> > more
> > > > > prominent
> > > > >
> > > > >
> > > > > Best,
> > > > > Chunjin Mu
> > > > >
> > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:26写道：
> > > > >
> > > > >> When a person asks which kafka should be used, I tell him in
> Chinese
> > > > that
> > > > >> he should use "采集整合的kafka", in fact the variable code is
> > > collect.kafka,
> > > > but
> > > > >> he is searching for gather keyword, it is difficult to just pass
> > > > >> collect.kafka The intuitive understanding is to "采集整合的kafka",
> > because
> > > > >> the information contained in the variable code is still not that
> > much.
> > > > >>
> > > > >> Huajie Wang <be...@gmail.com> 于2022年10月14日周五 22:17写道：
> > > > >>
> > > > >>> I just thought about it, there is really no need for this name.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> Best,
> > > > >>> Huajie Wang
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>> Rui Fan <19...@gmail.com> 于2022年10月14日周五 21:54写道：
> > > > >>>
> > > > >>> > Hi guys,
> > > > >>> >
> > > > >>> > I understand chunjin's solution is this:
> > > > >>> >
> > > > >>> > - variable code is placeholder
> > > > >>> > - variable name is an introduction
> > > > >>> > - Description is a detailed introduction
> > > > >>> >
> > > > >>> > I think introduction and detail are redundant.
> > > > >>> >
> > > > >>> > For example: there are two kafka clusters, the Beijing kafka
> > > > >>> > cluster and the Hangzhou kafka cluster.
> > > > >>> >
> > > > >>> > If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why
> > > > >>> > does the introduction still need `北京 kafka 集群` and
> > > > >>> > `杭州 kafka 集群`? placeholder is already clear.
> > > > >>> >
> > > > >>> > If dev may forget the placeholder, then dev may also
> > > > >>> > forget the introduction.
> > > > >>> >
> > > > >>> > Of course, if the placeholder is kafka1 kafka2, an
> > > > >>> > introduction is required. Why do we need an introduction
> > > > >>> > at this point, because the placeholder name is unreasonable.
> > > > >>> >
> > > > >>> > Just like variable naming in Java code, there is no need to
> > > > >>> > introduce an introduction with a clear name. If the meaning of
> > > > >>> > the variable is complicated, we also need to write a detailed
> > > > >>> > description.
> > > > >>> >
> > > > >>> > Finally, I want to say: less is more. If we want to make
> > > > >>> > something complicated, please give everyone a strong reason.
> > > > >>> >
> > > > >>> > Best
> > > > >>> > fanrui
> > > > >>> >
> > > > >>> > On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <benjobx@gmail.com
> >
> > > > wrote:
> > > > >>> >
> > > > >>> > > Hi guys,
> > > > >>> > >
> > > > >>> > > I agree with Chunjin, I think variable name is useful,
> Chunjin
> > > also
> > > > >>> > > explained the reason.
> > > > >>> > >
> > > > >>> > >
> > > > >>> > > Best,
> > > > >>> > > Huajie Wang
> > > > >>> > >
> > > > >>> > >
> > > > >>> > >
> > > > >>> > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 20:22写道：
> > > > >>> > >
> > > > >>> > > > Hi, Rui Fan
> > > > >>> > > > I'll start a discussion later.
> > > > >>> > > >
> > > > >>> > > > Best
> > > > >>> > > > Chunjin Mu
> > > > >>> > > >
> > > > >>> > > > Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
> > > > >>> > > >
> > > > >>> > > > > Hi guys,
> > > > >>> > > > >
> > > > >>> > > > > I agree with monster. And I suggest use variable name,
> > > variable
> > > > >>> value
> > > > >>> > > and
> > > > >>> > > > > description.
> > > > >>> > > > >
> > > > >>> > > > > By the way, there are some problems still need to
> confirm.
> > > > >>> What’s the
> > > > >>> > > > > placeholder? And how to use them when edit sql or
> > datastream
> > > > job?
> > > > >>> > It’s
> > > > >>> > > > > better to discuss these asap.
> > > > >>> > > > >
> > > > >>> > > > > Best
> > > > >>> > > > > fanrui
> > > > >>> > > > >
> > > > >>> > > > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <
> > > macksonmu0@gmail.com
> > > > >
> > > > >>> > wrote:
> > > > >>> > > > >
> > > > >>> > > > > > hi, huajie, fan rui, monster
> > > > >>> > > > > >
> > > > >>> > > > > > Thank you for your suggestions.
> > > > >>> > > > > >
> > > > >>> > > > > > Now please Huajie to confirm how to do it. If remove
> the
> > > > >>> variable
> > > > >>> > > > name, I
> > > > >>> > > > > > can also accept it here. I will do small secondary
> > > > development
> > > > >>> on
> > > > >>> > our
> > > > >>> > > > > > production system based on the community to meet
> internal
> > > > >>> needs.
> > > > >>> > > > > >
> > > > >>> > > > > > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五
> > 15:58写道：
> > > > >>> > > > > >
> > > > >>> > > > > > > "外部采集整合kafka" is the Chinese description of variable,
> > so
> > > it
> > > > >>> makes
> > > > >>> > > > sense
> > > > >>> > > > > > to
> > > > >>> > > > > > > put it under `Description`.
> > > > >>> > > > > > >
> > > > >>> > > > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五
> > > 12:01写道：
> > > > >>> > > > > > >
> > > > >>> > > > > > > > The variable management module has three
> properties,
> > > > >>> including
> > > > >>> > > > > variable
> > > > >>> > > > > > > > code, variable name, and description, For example,
> > the
> > > > >>> variable
> > > > >>> > > > code
> > > > >>> > > > > is
> > > > >>> > > > > > > > "external.kafka.collect.brokers", and the variable
> > name
> > > > is
> > > > >>> > > > > > "外部采集整合kafka".
> > > > >>> > > > > > > > When we created the application, I forgot what the
> > > > variable
> > > > >>> > code
> > > > >>> > > > was,
> > > > >>> > > > > > > but I
> > > > >>> > > > > > > > knew it was called "采集整合", so I can search Enter
> the
> > > > "采集整合"
> > > > >>> > > > keywords
> > > > >>> > > > > in
> > > > >>> > > > > > > the
> > > > >>> > > > > > > > box, and of course you can also search for kafka
> > > > keywords.
> > > > >>> > > > > > > >
> > > > >>> > > > > > > > At the same time, I want to know more detailed
> > > > information
> > > > >>> > about
> > > > >>> > > > this
> > > > >>> > > > > > > > kafka, such as who the operator is, what is the
> data
> > > > >>> retention
> > > > >>> > > > > period,
> > > > >>> > > > > > > and
> > > > >>> > > > > > > > how big is the cluster size, but I can't use the
> > > > >>> description to
> > > > >>> > > > > search,
> > > > >>> > > > > > > so
> > > > >>> > > > > > > > I can only search or variable code or variable
> name,
> > > and
> > > > >>> the
> > > > >>> > > > variable
> > > > >>> > > > > > > name
> > > > >>> > > > > > > > is also for a simple display.
> > > > >>> > > > > > > >
> > > > >>> > > > > > > > Although external.kafka.collect.brokers already
> > > contains
> > > > a
> > > > >>> lot
> > > > >>> > of
> > > > >>> > > > > > > > information, I think this is just a naming
> > convention.
> > > > >>> When we
> > > > >>> > > > > discuss
> > > > >>> > > > > > > this
> > > > >>> > > > > > > > variable, we will discuss it by using the "采集整合
> > kafka"
> > > > >>> > > > > > > >
> > > > >>> > > > > > > > We can discuss whether there is a problem with this
> > > > design,
> > > > >>> > what
> > > > >>> > > do
> > > > >>> > > > > you
> > > > >>> > > > > > > > think?
> > > > >>> > > > > > > >
> > > > >>> > > > > > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一
> > 11:45写道：
> > > > >>> > > > > > > >
> > > > >>> > > > > > > > > Hi huajie:
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > > > If it is manually entered, it will always be
> > > treated
> > > > as
> > > > >>> > > > ordinary
> > > > >>> > > > > > > text.
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > > Users always copy some code from an old job to a
> > new
> > > > job.
> > > > >>> > > > > > > > > I'm afraid it might not meet the user's
> > expectations.
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > > Best,
> > > > >>> > > > > > > > > fanrui
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <
> > > > >>> > benjobx@gmail.com>
> > > > >>> > > > > > wrote:
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > > > > hi 田向阳:
> > > > >>> > > > > > > > > >
> > > > >>> > > > > > > > > > The current discussion is about variable
> > > management,
> > > > >>> which
> > > > >>> > > is a
> > > > >>> > > > > > > > > relatively
> > > > >>> > > > > > > > > > independent function, No other modules
> involved,
> > > you
> > > > >>> can
> > > > >>> > > check
> > > > >>> > > > > the
> > > > >>> > > > > > > > thread
> > > > >>> > > > > > > > > > [1]
> > > > >>> > > > > > > > > >
> > > > >>> > > > > > > > > >
> > > > >>> > > > > > > > > > [1] thread:
> > > > >>> > > > > > > > > >
> > > > >>> > > >
> > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > > > >>> > > > > > > > > >
> > > > >>> > > > > > > > > >
> > > > >>> > > > > > > > > > Best,
> > > > >>> > > > > > > > > > Huajie Wang
> > > > >>> > > > > > > > > >
> > > > >>> > > > > > > > > >
> > > > >>> > > > > > > > > >
> > > > >>> > > > > > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > > > >>> > > > > > > > > >
> > > > >>> > > > > > > > > > > is this module include a resource management
> > > cente
> > > > r
> > > > >>> > where
> > > > >>> > > > jars
> > > > >>> > > > > > can
> > > > >>> > > > > > > > be
> > > > >>> > > > > > > > > > > chosen to provide to create a job
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > > ---- 回复的原邮件 ----
> > > > >>> > > > > > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > > >>> > > > > > > > > > > | 日期 | 2022年10月08日 12:07 |
> > > > >>> > > > > > > > > > > | 收件人 | dev@streampark.apache.org<
> > > > >>> > > dev@streampark.apache.org>
> > > > >>> > > > |
> > > > >>> > > > > > > > > > > | 抄送至 | |
> > > > >>> > > > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable
> > > > >>> management
> > > > >>> > > > > module |
> > > > >>> > > > > > > > > > > hi 阿穆:
> > > > >>> > > > > > > > > > > Thank you for the clarification, My question
> is
> > > > >>> answered,
> > > > >>> > > but
> > > > >>> > > > > > now I
> > > > >>> > > > > > > > > have
> > > > >>> > > > > > > > > > > one more concern,
> > > > >>> > > > > > > > > > > I'm currently using {var} for variables, and
> > I'm
> > > > >>> worried
> > > > >>> > > that
> > > > >>> > > > > one
> > > > >>> > > > > > > of
> > > > >>> > > > > > > > > the
> > > > >>> > > > > > > > > > > user's parameters includes "{" and "}", e.g :
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > > --password 'mypass{123}abc'
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > > So how do we parse parse this? How about
> > ${var}
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > > Best,
> > > > >>> > > > > > > > > > > Huajie Wang
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六
> > > > 11:36写道：
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > Thanks for your reply.
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > I have integrated your replies. I really
> > don't
> > > > pay
> > > > >>> much
> > > > >>> > > > > > attention
> > > > >>> > > > > > > > to
> > > > >>> > > > > > > > > > SQL,
> > > > >>> > > > > > > > > > > > so I will do the following.
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > 1.Each team has independent variables that
> do
> > > not
> > > > >>> > affect
> > > > >>> > > > each
> > > > >>> > > > > > > other
> > > > >>> > > > > > > > > and
> > > > >>> > > > > > > > > > > > are isolated from each other, but the same
> > > > >>> variable is
> > > > >>> > > > unique
> > > > >>> > > > > > > under
> > > > >>> > > > > > > > > the
> > > > >>> > > > > > > > > > > > same team, so you need to specify the team
> > when
> > > > >>> > creating
> > > > >>> > > > > > > variables
> > > > >>> > > > > > > > > > (users
> > > > >>> > > > > > > > > > > > may belong to multiple teams, and the teams
> > of
> > > > the
> > > > >>> > > current
> > > > >>> > > > > user
> > > > >>> > > > > > > > will
> > > > >>> > > > > > > > > be
> > > > >>> > > > > > > > > > > > displayed here for users to choose)
> > > > >>> > > > > > > > > > > > 2.For Flink jar jobs, there are two ways to
> > > pass
> > > > >>> > > variables
> > > > >>> > > > > > > > > > > > - By using placeholders, such as adding
> > --kafka
> > > > >>> > {mykafka}
> > > > >>> > > > to
> > > > >>> > > > > > the
> > > > >>> > > > > > > > args
> > > > >>> > > > > > > > > > of
> > > > >>> > > > > > > > > > > > the application, 'mykafka' is the variable
> > code
> > > > of
> > > > >>> a
> > > > >>> > > > variable
> > > > >>> > > > > > > > > > > > - Search by variable code or variable name.
> > > > After a
> > > > >>> > > > variable
> > > > >>> > > > > is
> > > > >>> > > > > > > > > > selected,
> > > > >>> > > > > > > > > > > > the parameter transfer method is displayed
> > > > >>> > > > > > > > > > > > 3.Flink sql will use variables as
> > placeholders,
> > > > >>> E.g:
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > CREATE TABLE KafkaTableA (
> > > > >>> > > > > > > > > > > > `user_id` BIGINT,
> > > > >>> > > > > > > > > > > > `item_id` BIGINT,
> > > > >>> > > > > > > > > > > > `type` STRING
> > > > >>> > > > > > > > > > > > ) WITH (
> > > > >>> > > > > > > > > > > > 'connector' = 'kafka',
> > > > >>> > > > > > > > > > > > 'topic' = 'user_behaviorA',
> > > > >>> > > > > > > > > > > >
> 'properties.bootstrap.servers' =
> > > > >>> > > > > > > > > '{kafka_clusterA}',
> > > > >>> > > > > > > > > > > --
> > > > >>> > > > > > > > > > > > 'kafka_ ClusterA' is the variable code
> > > > >>> > > > > > > > > > > > 'properties.group.id' =
> > > > 'testGroupA',
> > > > >>> > > > > > > > > > > > 'scan.startup.mode' =
> > > > >>> 'earliest-offset',
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > CREATE TABLE KafkaTableB (
> > > > >>> > > > > > > > > > > > `user_id` BIGINT,
> > > > >>> > > > > > > > > > > > `item_id` BIGINT
> > > > >>> > > > > > > > > > > > ) WITH (
> > > > >>> > > > > > > > > > > > 'connector' = 'kafka',
> > > > >>> > > > > > > > > > > > 'topic' = 'user_behaviorB',
> > > > >>> > > > > > > > > > > >
> 'properties.bootstrap.servers' =
> > > > >>> > > > > > > > > '{kafka_clusterB}',
> > > > >>> > > > > > > > > > > --
> > > > >>> > > > > > > > > > > > 'kafka_ ClusterB' is the variable code
> > > > >>> > > > > > > > > > > > 'properties.group.id' =
> > > > 'testGroupB',
> > > > >>> > > > > > > > > > > > 'scan.startup.mode' =
> > > > >>> 'earliest-offset',
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id,
> > > item_id,
> > > > >>> > '{type}'
> > > > >>> > > > > FROM
> > > > >>> > > > > > > > > > > > KafkaTableB -- 'type' is the
> > > variable
> > > > >>> code
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > 4.In the future, we can consider setting
> some
> > > > >>> system
> > > > >>> > > > built-in
> > > > >>> > > > > > > > > variables
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > Best,
> > > > >>> > > > > > > > > > > > ChunJin Mu
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
> > > > >>> > > > 1996fanrui@gmail.com
> > > > >>> > > > > > >
> > > > >>> > > > > > > > > > wrote:
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > Hi 阿穆:
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > It's a great proposal.
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > For usage scenarios, I strongly
> > recommend
> > > > >>> support
> > > > >>> > > for
> > > > >>> > > > > > > > replacing
> > > > >>> > > > > > > > > > > > variables
> > > > >>> > > > > > > > > > > > > in sql. E.g:
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > CREATE TABLE KafkaTable (
> > > > >>> > > > > > > > > > > > > `user_id` BIGINT,
> > > > >>> > > > > > > > > > > > > `item_id` BIGINT
> > > > >>> > > > > > > > > > > > > ) WITH (
> > > > >>> > > > > > > > > > > > > 'connector' =
> 'kafka',
> > > > >>> > > > > > > > > > > > > 'topic' =
> > > 'user_behavior',
> > > > >>> > > > > > > > > > > > >
> > > > >>> 'properties.bootstrap.servers' =
> > > > >>> > > > > > > > > > > '{kafka_clusterA}',
> > > > >>> > > > > > > > > > > > > '
> properties.group.id'
> > =
> > > > >>> > > > 'testGroup',
> > > > >>> > > > > > > > > > > > > 'scan.startup.mode'
> =
> > > > >>> > > > > 'earliest-offset',
> > > > >>> > > > > > > > > > > > > 'format' = 'csv'
> > > > >>> > > > > > > > > > > > > )
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > Note: 'properties.bootstrap.servers' =
> > > > >>> > > > > '{kafka_clusterA}',
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > For variable management, I have some
> > > > question:
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > 1. Is the parameter used within the
> > team?
> > > If
> > > > >>> all
> > > > >>> > > teams
> > > > >>> > > > > > use a
> > > > >>> > > > > > > > > > common
> > > > >>> > > > > > > > > > > > > parameter, it
> > > > >>> > > > > > > > > > > > > needs to be defined for each team,
> > right?
> > > > >>> > > > > > > > > > > > > 2. Which users can modify variables?
> Can
> > > all
> > > > >>> users
> > > > >>> > > > > modify
> > > > >>> > > > > > > all
> > > > >>> > > > > > > > > > > > variables of
> > > > >>> > > > > > > > > > > > > their team?
> > > > >>> > > > > > > > > > > > > After the variable is modified
> > > incorrectly,
> > > > >>> many
> > > > >>> > > jobs
> > > > >>> > > > > may
> > > > >>> > > > > > be
> > > > >>> > > > > > > > > > > > affected. So I
> > > > >>> > > > > > > > > > > > > suggest that
> > > > >>> > > > > > > > > > > > > only team admin or system admin have
> > > > >>> permission to
> > > > >>> > > > > modify,
> > > > >>> > > > > > > add
> > > > >>> > > > > > > > > and
> > > > >>> > > > > > > > > > > > delete
> > > > >>> > > > > > > > > > > > > variables.
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > Best
> > > > >>> > > > > > > > > > > > > fanrui
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang
> <
> > > > >>> > > > > benjobx@gmail.com
> > > > >>> > > > > > > >
> > > > >>> > > > > > > > > > wrote:
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > hi 阿穆:
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > Thanks for starting a very useful
> > > > discussion,
> > > > >>> I
> > > > >>> > > have a
> > > > >>> > > > > > some
> > > > >>> > > > > > > > > > > questions:
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > 1. The access boundary problem of
> > variable
> > > > >>> > > management,
> > > > >>> > > > > is
> > > > >>> > > > > > > this
> > > > >>> > > > > > > > > all
> > > > >>> > > > > > > > > > > > > people in the platform can access
> these
> > > > >>> variables(
> > > > >>> > > no
> > > > >>> > > > > > matter
> > > > >>> > > > > > > > who
> > > > >>> > > > > > > > > > > > > defines) or that only team members can
> > be
> > > > >>> access
> > > > >>> > > these
> > > > >>> > > > > > > > > variables.
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > 2. There are two usage scenarios for
> > > > variables
> > > > >>> > that
> > > > >>> > > I
> > > > >>> > > > > know
> > > > >>> > > > > > > of,
> > > > >>> > > > > > > > > one
> > > > >>> > > > > > > > > > > is
> > > > >>> > > > > > > > > > > > > in the entry parameter of main, and
> the
> > > > other
> > > > >>> is
> > > > >>> > in
> > > > >>> > > > > > > flinksql.
> > > > >>> > > > > > > > > Are
> > > > >>> > > > > > > > > > > both
> > > > >>> > > > > > > > > > > > > supported, or only supported main
> entry
> > ?
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > about first question, my opinion:
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > 1. The variable management is
> associated
> > > > with
> > > > >>> the
> > > > >>> > > > team.
> > > > >>> > > > > > > After
> > > > >>> > > > > > > > > the
> > > > >>> > > > > > > > > > > user
> > > > >>> > > > > > > > > > > > > logs in to the system, he must select
> a
> > > team
> > > > >>> (a
> > > > >>> > user
> > > > >>> > > > can
> > > > >>> > > > > > > > belong
> > > > >>> > > > > > > > > to
> > > > >>> > > > > > > > > > > one
> > > > >>> > > > > > > > > > > > > or more teams), The variable belongs
> to
> > a
> > > > team
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > 2. All resources between teams are
> > > isolated,
> > > > >>> which
> > > > >>> > > > means
> > > > >>> > > > > > > that
> > > > >>> > > > > > > > > > > > > variables are also isolated. In short,
> > > > >>> members of
> > > > >>> > a
> > > > >>> > > > team
> > > > >>> > > > > > can
> > > > >>> > > > > > > > use
> > > > >>> > > > > > > > > > and
> > > > >>> > > > > > > > > > > > > define these variables, which are
> > > invisible
> > > > to
> > > > >>> > other
> > > > >>> > > > > > > members,
> > > > >>> > > > > > > > > and
> > > > >>> > > > > > > > > > > > > other members cannot use these
> > variables.
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > Welcome to your views
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > Best,
> > > > >>> > > > > > > > > > > > > Huajie Wang
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com>
> > > > >>> > 于2022年10月7日周五
> > > > >>> > > > > > > 14:18写道：
> > > > >>> > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > Is it possible to do this
> variable
> > > > >>> support in
> > > > >>> > > sql
> > > > >>> > > > > as
> > > > >>> > > > > > > well
> > > > >>> > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > 阿穆 <305343894@qq.com.invalid>
> > > > >>> > 于2022年10月7日周五
> > > > >>> > > > > > > 13:54写道：
> > > > >>> > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > > Hi guys,
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > > I want to start a discussion
> > > about
> > > > >>> the
> > > > >>> > > > > variable
> > > > >>> > > > > > > > > > management
> > > > >>> > > > > > > > > > > > module. Please
> > > > >>> > > > > > > > > > > > > > > see this issue issue-1477.
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > > Background:
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > > In the actual production
> > > > >>> environment,
> > > > >>> > > Flink
> > > > >>> > > > > jobs
> > > > >>> > > > > > > are
> > > > >>> > > > > > > > > > > > generally complex,
> > > > >>> > > > > > > > > > > > > > > and usually require multiple
> > > > >>> external
> > > > >>> > > > > > components.
> > > > >>> > > > > > > > For
> > > > >>> > > > > > > > > > > > example, Flink jobs
> > > > >>> > > > > > > > > > > > > > > consume data from Kafka,
> then
> > > > >>> connect
> > > > >>> > > > external
> > > > >>> > > > > > > > > > components
> > > > >>> > > > > > > > > > > > such as HBase
> > > > >>> > > > > > > > > > > > > > or
> > > > >>> > > > > > > > > > > > > > > Redis to obtain additional
> > > > business
> > > > >>> > > > > information,
> > > > >>> > > > > > > and
> > > > >>> > > > > > > > > > then
> > > > >>> > > > > > > > > > > > write it to the
> > > > >>> > > > > > > > > > > > > > > downstream external
> > components.
> > > > >>> There
> > > > >>> > are
> > > > >>> > > > the
> > > > >>> > > > > > > > > following
> > > > >>> > > > > > > > > > > > problems:
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > > 1. The connection
> information
> > of
> > > > >>> > external
> > > > >>> > > > > > > > components,
> > > > >>> > > > > > > > > > such
> > > > >>> > > > > > > > > > > > as IP, port
> > > > >>> > > > > > > > > > > > > > and
> > > > >>> > > > > > > > > > > > > > > user password, needs to be
> > > > >>> configured in
> > > > >>> > > the
> > > > >>> > > > > > > > > application
> > > > >>> > > > > > > > > > > > args and
> > > > >>> > > > > > > > > > > > > > > transferred to the Flink
> job,
> > so
> > > > >>> that
> > > > >>> > the
> > > > >>> > > > > > > connection
> > > > >>> > > > > > > > > > > > information of
> > > > >>> > > > > > > > > > > > > > > external components is
> > > distributed
> > > > >>> in
> > > > >>> > > > multiple
> > > > >>> > > > > > > > > > > > applications. Once the
> > > > >>> > > > > > > > > > > > > > > connection information of
> > > external
> > > > >>> > > > components
> > > > >>> > > > > > > > changes,
> > > > >>> > > > > > > > > > > many
> > > > >>> > > > > > > > > > > > application
> > > > >>> > > > > > > > > > > > > > > args parameters need to be
> > > > modified,
> > > > >>> > which
> > > > >>> > > > > will
> > > > >>> > > > > > > lead
> > > > >>> > > > > > > > > to
> > > > >>> > > > > > > > > > > > high operation
> > > > >>> > > > > > > > > > > > > > and
> > > > >>> > > > > > > > > > > > > > > maintenance costs.
> > > > >>> > > > > > > > > > > > > > > 2. There are many people in
> > the
> > > > team
> > > > >>> > > > > developing
> > > > >>> > > > > > > > > > real-time
> > > > >>> > > > > > > > > > > > computing jobs.
> > > > >>> > > > > > > > > > > > > > > There is no uniform
> > > specification
> > > > >>> for
> > > > >>> > the
> > > > >>> > > > > > > connection
> > > > >>> > > > > > > > > > > > information of the
> > > > >>> > > > > > > > > > > > > > > external components passed
> to
> > > the
> > > > >>> job,
> > > > >>> > > > > resulting
> > > > >>> > > > > > > in
> > > > >>> > > > > > > > > > > > different parameter
> > > > >>> > > > > > > > > > > > > > > names of the same component.
> > > This
> > > > is
> > > > >>> > > > difficult
> > > > >>> > > > > > to
> > > > >>> > > > > > > > > count
> > > > >>> > > > > > > > > > > > which external
> > > > >>> > > > > > > > > > > > > > > components are dependent.
> > > > >>> > > > > > > > > > > > > > > 3. In production practice,
> > there
> > > > are
> > > > >>> > > usually
> > > > >>> > > > > > > > multiple
> > > > >>> > > > > > > > > > sets
> > > > >>> > > > > > > > > > > > of
> > > > >>> > > > > > > > > > > > > > > environments, such as test
> > > > >>> environment
> > > > >>> > and
> > > > >>> > > > > > > > production
> > > > >>> > > > > > > > > > > > environment. It is
> > > > >>> > > > > > > > > > > > > > > not intuitive to judge
> > whether a
> > > > >>> > component
> > > > >>> > > > > > belongs
> > > > >>> > > > > > > > to
> > > > >>> > > > > > > > > a
> > > > >>> > > > > > > > > > > > test environment
> > > > >>> > > > > > > > > > > > > > or
> > > > >>> > > > > > > > > > > > > > > a production environment
> > simply
> > > > >>> through
> > > > >>> > IP
> > > > >>> > > > and
> > > > >>> > > > > > > > ports.
> > > > >>> > > > > > > > > > Even
> > > > >>> > > > > > > > > > > > if it can be
> > > > >>> > > > > > > > > > > > > > > judged, there are some
> > > omissions.
> > > > In
> > > > >>> > this
> > > > >>> > > > way,
> > > > >>> > > > > > the
> > > > >>> > > > > > > > > > > > connection information
> > > > >>> > > > > > > > > > > > > > > online to the production
> > > > >>> environment may
> > > > >>> > > be
> > > > >>> > > > > > > external
> > > > >>> > > > > > > > > > > > components of the
> > > > >>> > > > > > > > > > > > > > test
> > > > >>> > > > > > > > > > > > > > > environment, or components
> of
> > > the
> > > > >>> > > production
> > > > >>> > > > > > > > > environment
> > > > >>> > > > > > > > > > > are
> > > > >>> > > > > > > > > > > > > > inadvertently
> > > > >>> > > > > > > > > > > > > > > configured during testing,
> > > leading
> > > > >>> to
> > > > >>> > > > > production
> > > > >>> > > > > > > > > > failures.
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > > Proposal:
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > > 1. Create a unified module
> to
> > > > manage
> > > > >>> > > > > variables.
> > > > >>> > > > > > > > Users
> > > > >>> > > > > > > > > > can
> > > > >>> > > > > > > > > > > > define their
> > > > >>> > > > > > > > > > > > > > own
> > > > >>> > > > > > > > > > > > > > > variables in this module.
> > > Variable
> > > > >>> > > > information
> > > > >>> > > > > > > > > includes
> > > > >>> > > > > > > > > > > > variable code
> > > > >>> > > > > > > > > > > > > > > (passed to Flink program as
> > > > >>> parameter
> > > > >>> > > name),
> > > > >>> > > > > > > > variable
> > > > >>> > > > > > > > > > > > value, variable
> > > > >>> > > > > > > > > > > > > name,
> > > > >>> > > > > > > > > > > > > > > variable description, and
> > which
> > > > >>> > > applications
> > > > >>> > > > > > > depend
> > > > >>> > > > > > > > on
> > > > >>> > > > > > > > > > > > > > > 2. In the same Team module,
> > the
> > > > >>> variable
> > > > >>> > > > code
> > > > >>> > > > > or
> > > > >>> > > > > > > > > > variable
> > > > >>> > > > > > > > > > > > name is unique
> > > > >>> > > > > > > > > > > > > > > - By using placeholders,
> such
> > as
> > > > >>> adding
> > > > >>> > --
> > > > >>> > > > > kafka
> > > > >>> > > > > > > > > > {mykafka}
> > > > >>> > > > > > > > > > > > to the args of
> > > > >>> > > > > > > > > > > > > > > the application, mykafka is
> > the
> > > > >>> variable
> > > > >>> > > > code
> > > > >>> > > > > > of a
> > > > >>> > > > > > > > > > > variable
> > > > >>> > > > > > > > > > > > > > > - Search by variable code or
> > > > >>> variable
> > > > >>> > > name.
> > > > >>> > > > > > After
> > > > >>> > > > > > > a
> > > > >>> > > > > > > > > > > > variable is selected,
> > > > >>> > > > > > > > > > > > > > > the parameter transfer
> method
> > is
> > > > >>> > displayed
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > > If you have any suggestions,
> > > > >>> welcome to
> > > > >>> > > > > > > participate
> > > > >>> > > > > > > > in
> > > > >>> > > > > > > > > > the
> > > > >>> > > > > > > > > > > > discussion on
> > > > >>> > > > > > > > > > > > > > > the mail list or issue, I
> look
> > > > >>> forward
> > > > >>> > to
> > > > >>> > > > your
> > > > >>> > > > > > > > > feedback.
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > > > > > > Best wishes
> > > > >>> > > > > > > > > > > > > >
> > > > >>> > > > > > > > > > >
> > > > >>> > > > > > > > > >
> > > > >>> > > > > > > > >
> > > > >>> > > > > > > >
> > > > >>> > > > > > >
> > > > >>> > > > > >
> > > > >>> > > > >
> > > > >>> > > >
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > >
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Rui Fan <19...@gmail.com>.
Hi chunjin,
Very sorry for slowing down your progress, I found some
small problems when I reviewed your code.
Who can create the variable and update the variable?
System Admin and team_admin? Or every developer?
As I understand, we didn't discuss this point, right? If I
miss any information please let me know, thanks a lot.
Best,
fanrui
On Sat, Oct 15, 2022 at 4:44 PM Chunjin Mu <ma...@gmail.com> wrote:
> Hi fanru,
>
> > I have a little question. If I copy some sql from a old app, and the sql
> includes some placeholders. These placeholders will take effect in the
> new app, right?
>
> yes, the copied placeholder will take effect if the variable exists.
>
> I have added all the information to proposal issue-1477[1]，include the
> questions you asked above， if there is more discussion later, I will add
> this to the proposal.
>
> [1] https://github.com/apache/incubator-streampark/issues/1477
>
>
> Rui Fan <19...@gmail.com> 于2022年10月15日周六 13:17写道：
>
> > Hi chunjin,
> >
> > Thanks for your detailed information.
> >
> > > When the focus is lost, it is judged whether there is a variable
> > placeholder, and if so, a verification button is displayed to prompt the
> > user to perform verification. When the user clicks the verification
> button,
> > the replaced content of the placeholder is displayed, and only the user
> > clicks the verification button to click the submit button
> >
> > I have a little question. If I copy some sql from a old app, and the sql
> > includes some placeholders. These placeholders will take effect in the
> > new app, right?
> >
> > BTW, could you add all information to your proposal issue-1477[1]?
> > The mail list is for discussion only, and we should maintain the final
> > solution in the Proposal.
> >
> >
> > [1] https://github.com/apache/incubator-streampark/issues/1477
> >
> > Best
> > fanrui
> >
> > On Sat, Oct 15, 2022 at 11:56 AM Chunjin Mu <ma...@gmail.com>
> wrote:
> >
> > > Hi guys,
> > >
> > > Thanks for the discussion and great suggestions.
> > >
> > > Here I summarize the functions to be completed in version 1.2.4
> > >
> > > 1.The table structure of the variable contains the following main
> > contents
> > > - variable code is placeholder, For example the variable code is
> > > "collect.kafka.cluster", it will be referenced as
> > ${collect.kafka.cluster},
> > > it is unique within a team
> > > - variable name is removed, after discussion, it is not necessarily
> > > retained
> > > - variable value is the value of a specific variable, such as ip and
> port
> > > - description is a detailed introduction
> > > - team is the team the variable belongs to
> > >
> > > 2.Basic functions of variable management
> > > - Add and modify variables: need to check whether the variable code is
> > > unique under the current team
> > > - Deleting a variable: It is necessary to check whether it has been
> > > referenced by the Application under the team to which the variable
> > belongs.
> > > If there is metadata management in the future, it is also necessary to
> > > check whether it has been referenced by the metadata. If it is
> > referenced,
> > > it will prompt that the variable is referenced and cannot be deleted.
> > >
> > > 3.References in Flink sql and program args
> > > - In the application, the variables are uniformly referenced in the
> form
> > of
> > > placeholders ${xxx}, such as in Flink sql, program args
> > > - For a better experience, it is necessary to provide the function of
> > > following the search prompt when entering variable placeholders. For
> > > example, when entering "${", the variable list is displayed, and when
> > > "${kafka" is entered, the display contains variables for kafka strings,
> > in
> > > "variable code" or "description".
> > > - When the focus is lost, it is judged whether there is a variable
> > > placeholder, and if so, a verification button is displayed to prompt
> the
> > > user to perform verification. When the user clicks the verification
> > button,
> > > the replaced content of the placeholder is displayed, and only the user
> > > clicks the verification button to click the submit button
> > > - If the user enters "okir23${j2Rts}pEt", this placeholder "${j2Rts}"
> > will
> > > not be replaced if "j2Rts" is not a variable we defined
> > >
> > > 4.Future plan
> > > - Display the application or metadata list of a referenced variable in
> > the
> > > variable basic management module
> > > - Improve user experience
> > >
> > > Best,
> > > Chunjin Mu
> > >
> > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:28写道：
> > >
> > > > When we have dozens or dozens of variables, this phenomenon may be
> more
> > > > prominent
> > > >
> > > >
> > > > Best,
> > > > Chunjin Mu
> > > >
> > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:26写道：
> > > >
> > > >> When a person asks which kafka should be used, I tell him in Chinese
> > > that
> > > >> he should use "采集整合的kafka", in fact the variable code is
> > collect.kafka,
> > > but
> > > >> he is searching for gather keyword, it is difficult to just pass
> > > >> collect.kafka The intuitive understanding is to "采集整合的kafka",
> because
> > > >> the information contained in the variable code is still not that
> much.
> > > >>
> > > >> Huajie Wang <be...@gmail.com> 于2022年10月14日周五 22:17写道：
> > > >>
> > > >>> I just thought about it, there is really no need for this name.
> > > >>>
> > > >>>
> > > >>>
> > > >>> Best,
> > > >>> Huajie Wang
> > > >>>
> > > >>>
> > > >>>
> > > >>> Rui Fan <19...@gmail.com> 于2022年10月14日周五 21:54写道：
> > > >>>
> > > >>> > Hi guys,
> > > >>> >
> > > >>> > I understand chunjin's solution is this:
> > > >>> >
> > > >>> > - variable code is placeholder
> > > >>> > - variable name is an introduction
> > > >>> > - Description is a detailed introduction
> > > >>> >
> > > >>> > I think introduction and detail are redundant.
> > > >>> >
> > > >>> > For example: there are two kafka clusters, the Beijing kafka
> > > >>> > cluster and the Hangzhou kafka cluster.
> > > >>> >
> > > >>> > If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why
> > > >>> > does the introduction still need `北京 kafka 集群` and
> > > >>> > `杭州 kafka 集群`? placeholder is already clear.
> > > >>> >
> > > >>> > If dev may forget the placeholder, then dev may also
> > > >>> > forget the introduction.
> > > >>> >
> > > >>> > Of course, if the placeholder is kafka1 kafka2, an
> > > >>> > introduction is required. Why do we need an introduction
> > > >>> > at this point, because the placeholder name is unreasonable.
> > > >>> >
> > > >>> > Just like variable naming in Java code, there is no need to
> > > >>> > introduce an introduction with a clear name. If the meaning of
> > > >>> > the variable is complicated, we also need to write a detailed
> > > >>> > description.
> > > >>> >
> > > >>> > Finally, I want to say: less is more. If we want to make
> > > >>> > something complicated, please give everyone a strong reason.
> > > >>> >
> > > >>> > Best
> > > >>> > fanrui
> > > >>> >
> > > >>> > On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <be...@gmail.com>
> > > wrote:
> > > >>> >
> > > >>> > > Hi guys,
> > > >>> > >
> > > >>> > > I agree with Chunjin, I think variable name is useful, Chunjin
> > also
> > > >>> > > explained the reason.
> > > >>> > >
> > > >>> > >
> > > >>> > > Best,
> > > >>> > > Huajie Wang
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 20:22写道：
> > > >>> > >
> > > >>> > > > Hi, Rui Fan
> > > >>> > > > I'll start a discussion later.
> > > >>> > > >
> > > >>> > > > Best
> > > >>> > > > Chunjin Mu
> > > >>> > > >
> > > >>> > > > Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
> > > >>> > > >
> > > >>> > > > > Hi guys,
> > > >>> > > > >
> > > >>> > > > > I agree with monster. And I suggest use variable name,
> > variable
> > > >>> value
> > > >>> > > and
> > > >>> > > > > description.
> > > >>> > > > >
> > > >>> > > > > By the way, there are some problems still need to confirm.
> > > >>> What’s the
> > > >>> > > > > placeholder? And how to use them when edit sql or
> datastream
> > > job?
> > > >>> > It’s
> > > >>> > > > > better to discuss these asap.
> > > >>> > > > >
> > > >>> > > > > Best
> > > >>> > > > > fanrui
> > > >>> > > > >
> > > >>> > > > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <
> > macksonmu0@gmail.com
> > > >
> > > >>> > wrote:
> > > >>> > > > >
> > > >>> > > > > > hi, huajie, fan rui, monster
> > > >>> > > > > >
> > > >>> > > > > > Thank you for your suggestions.
> > > >>> > > > > >
> > > >>> > > > > > Now please Huajie to confirm how to do it. If remove the
> > > >>> variable
> > > >>> > > > name, I
> > > >>> > > > > > can also accept it here. I will do small secondary
> > > development
> > > >>> on
> > > >>> > our
> > > >>> > > > > > production system based on the community to meet internal
> > > >>> needs.
> > > >>> > > > > >
> > > >>> > > > > > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五
> 15:58写道：
> > > >>> > > > > >
> > > >>> > > > > > > "外部采集整合kafka" is the Chinese description of variable,
> so
> > it
> > > >>> makes
> > > >>> > > > sense
> > > >>> > > > > > to
> > > >>> > > > > > > put it under `Description`.
> > > >>> > > > > > >
> > > >>> > > > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五
> > 12:01写道：
> > > >>> > > > > > >
> > > >>> > > > > > > > The variable management module has three properties,
> > > >>> including
> > > >>> > > > > variable
> > > >>> > > > > > > > code, variable name, and description, For example,
> the
> > > >>> variable
> > > >>> > > > code
> > > >>> > > > > is
> > > >>> > > > > > > > "external.kafka.collect.brokers", and the variable
> name
> > > is
> > > >>> > > > > > "外部采集整合kafka".
> > > >>> > > > > > > > When we created the application, I forgot what the
> > > variable
> > > >>> > code
> > > >>> > > > was,
> > > >>> > > > > > > but I
> > > >>> > > > > > > > knew it was called "采集整合", so I can search Enter the
> > > "采集整合"
> > > >>> > > > keywords
> > > >>> > > > > in
> > > >>> > > > > > > the
> > > >>> > > > > > > > box, and of course you can also search for kafka
> > > keywords.
> > > >>> > > > > > > >
> > > >>> > > > > > > > At the same time, I want to know more detailed
> > > information
> > > >>> > about
> > > >>> > > > this
> > > >>> > > > > > > > kafka, such as who the operator is, what is the data
> > > >>> retention
> > > >>> > > > > period,
> > > >>> > > > > > > and
> > > >>> > > > > > > > how big is the cluster size, but I can't use the
> > > >>> description to
> > > >>> > > > > search,
> > > >>> > > > > > > so
> > > >>> > > > > > > > I can only search or variable code or variable name,
> > and
> > > >>> the
> > > >>> > > > variable
> > > >>> > > > > > > name
> > > >>> > > > > > > > is also for a simple display.
> > > >>> > > > > > > >
> > > >>> > > > > > > > Although external.kafka.collect.brokers already
> > contains
> > > a
> > > >>> lot
> > > >>> > of
> > > >>> > > > > > > > information, I think this is just a naming
> convention.
> > > >>> When we
> > > >>> > > > > discuss
> > > >>> > > > > > > this
> > > >>> > > > > > > > variable, we will discuss it by using the "采集整合
> kafka"
> > > >>> > > > > > > >
> > > >>> > > > > > > > We can discuss whether there is a problem with this
> > > design,
> > > >>> > what
> > > >>> > > do
> > > >>> > > > > you
> > > >>> > > > > > > > think?
> > > >>> > > > > > > >
> > > >>> > > > > > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一
> 11:45写道：
> > > >>> > > > > > > >
> > > >>> > > > > > > > > Hi huajie:
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > > If it is manually entered, it will always be
> > treated
> > > as
> > > >>> > > > ordinary
> > > >>> > > > > > > text.
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > Users always copy some code from an old job to a
> new
> > > job.
> > > >>> > > > > > > > > I'm afraid it might not meet the user's
> expectations.
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > Best,
> > > >>> > > > > > > > > fanrui
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <
> > > >>> > benjobx@gmail.com>
> > > >>> > > > > > wrote:
> > > >>> > > > > > > > >
> > > >>> > > > > > > > > > hi 田向阳:
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > The current discussion is about variable
> > management,
> > > >>> which
> > > >>> > > is a
> > > >>> > > > > > > > > relatively
> > > >>> > > > > > > > > > independent function, No other modules involved,
> > you
> > > >>> can
> > > >>> > > check
> > > >>> > > > > the
> > > >>> > > > > > > > thread
> > > >>> > > > > > > > > > [1]
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > [1] thread:
> > > >>> > > > > > > > > >
> > > >>> > > >
> > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > Best,
> > > >>> > > > > > > > > > Huajie Wang
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > > > > is this module include a resource management
> > cente
> > > r
> > > >>> > where
> > > >>> > > > jars
> > > >>> > > > > > can
> > > >>> > > > > > > > be
> > > >>> > > > > > > > > > > chosen to provide to create a job
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > ---- 回复的原邮件 ----
> > > >>> > > > > > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > >>> > > > > > > > > > > | 日期 | 2022年10月08日 12:07 |
> > > >>> > > > > > > > > > > | 收件人 | dev@streampark.apache.org<
> > > >>> > > dev@streampark.apache.org>
> > > >>> > > > |
> > > >>> > > > > > > > > > > | 抄送至 | |
> > > >>> > > > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable
> > > >>> management
> > > >>> > > > > module |
> > > >>> > > > > > > > > > > hi 阿穆:
> > > >>> > > > > > > > > > > Thank you for the clarification, My question is
> > > >>> answered,
> > > >>> > > but
> > > >>> > > > > > now I
> > > >>> > > > > > > > > have
> > > >>> > > > > > > > > > > one more concern,
> > > >>> > > > > > > > > > > I'm currently using {var} for variables, and
> I'm
> > > >>> worried
> > > >>> > > that
> > > >>> > > > > one
> > > >>> > > > > > > of
> > > >>> > > > > > > > > the
> > > >>> > > > > > > > > > > user's parameters includes "{" and "}", e.g :
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > --password 'mypass{123}abc'
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > So how do we parse parse this? How about
> ${var}
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > Best,
> > > >>> > > > > > > > > > > Huajie Wang
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六
> > > 11:36写道：
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > Thanks for your reply.
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > I have integrated your replies. I really
> don't
> > > pay
> > > >>> much
> > > >>> > > > > > attention
> > > >>> > > > > > > > to
> > > >>> > > > > > > > > > SQL,
> > > >>> > > > > > > > > > > > so I will do the following.
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > 1.Each team has independent variables that do
> > not
> > > >>> > affect
> > > >>> > > > each
> > > >>> > > > > > > other
> > > >>> > > > > > > > > and
> > > >>> > > > > > > > > > > > are isolated from each other, but the same
> > > >>> variable is
> > > >>> > > > unique
> > > >>> > > > > > > under
> > > >>> > > > > > > > > the
> > > >>> > > > > > > > > > > > same team, so you need to specify the team
> when
> > > >>> > creating
> > > >>> > > > > > > variables
> > > >>> > > > > > > > > > (users
> > > >>> > > > > > > > > > > > may belong to multiple teams, and the teams
> of
> > > the
> > > >>> > > current
> > > >>> > > > > user
> > > >>> > > > > > > > will
> > > >>> > > > > > > > > be
> > > >>> > > > > > > > > > > > displayed here for users to choose)
> > > >>> > > > > > > > > > > > 2.For Flink jar jobs, there are two ways to
> > pass
> > > >>> > > variables
> > > >>> > > > > > > > > > > > - By using placeholders, such as adding
> --kafka
> > > >>> > {mykafka}
> > > >>> > > > to
> > > >>> > > > > > the
> > > >>> > > > > > > > args
> > > >>> > > > > > > > > > of
> > > >>> > > > > > > > > > > > the application, 'mykafka' is the variable
> code
> > > of
> > > >>> a
> > > >>> > > > variable
> > > >>> > > > > > > > > > > > - Search by variable code or variable name.
> > > After a
> > > >>> > > > variable
> > > >>> > > > > is
> > > >>> > > > > > > > > > selected,
> > > >>> > > > > > > > > > > > the parameter transfer method is displayed
> > > >>> > > > > > > > > > > > 3.Flink sql will use variables as
> placeholders,
> > > >>> E.g:
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > CREATE TABLE KafkaTableA (
> > > >>> > > > > > > > > > > > `user_id` BIGINT,
> > > >>> > > > > > > > > > > > `item_id` BIGINT,
> > > >>> > > > > > > > > > > > `type` STRING
> > > >>> > > > > > > > > > > > ) WITH (
> > > >>> > > > > > > > > > > > 'connector' = 'kafka',
> > > >>> > > > > > > > > > > > 'topic' = 'user_behaviorA',
> > > >>> > > > > > > > > > > > 'properties.bootstrap.servers' =
> > > >>> > > > > > > > > '{kafka_clusterA}',
> > > >>> > > > > > > > > > > --
> > > >>> > > > > > > > > > > > 'kafka_ ClusterA' is the variable code
> > > >>> > > > > > > > > > > > 'properties.group.id' =
> > > 'testGroupA',
> > > >>> > > > > > > > > > > > 'scan.startup.mode' =
> > > >>> 'earliest-offset',
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > CREATE TABLE KafkaTableB (
> > > >>> > > > > > > > > > > > `user_id` BIGINT,
> > > >>> > > > > > > > > > > > `item_id` BIGINT
> > > >>> > > > > > > > > > > > ) WITH (
> > > >>> > > > > > > > > > > > 'connector' = 'kafka',
> > > >>> > > > > > > > > > > > 'topic' = 'user_behaviorB',
> > > >>> > > > > > > > > > > > 'properties.bootstrap.servers' =
> > > >>> > > > > > > > > '{kafka_clusterB}',
> > > >>> > > > > > > > > > > --
> > > >>> > > > > > > > > > > > 'kafka_ ClusterB' is the variable code
> > > >>> > > > > > > > > > > > 'properties.group.id' =
> > > 'testGroupB',
> > > >>> > > > > > > > > > > > 'scan.startup.mode' =
> > > >>> 'earliest-offset',
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id,
> > item_id,
> > > >>> > '{type}'
> > > >>> > > > > FROM
> > > >>> > > > > > > > > > > > KafkaTableB -- 'type' is the
> > variable
> > > >>> code
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > 4.In the future, we can consider setting some
> > > >>> system
> > > >>> > > > built-in
> > > >>> > > > > > > > > variables
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > Best,
> > > >>> > > > > > > > > > > > ChunJin Mu
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
> > > >>> > > > 1996fanrui@gmail.com
> > > >>> > > > > > >
> > > >>> > > > > > > > > > wrote:
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > Hi 阿穆:
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > It's a great proposal.
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > For usage scenarios, I strongly
> recommend
> > > >>> support
> > > >>> > > for
> > > >>> > > > > > > > replacing
> > > >>> > > > > > > > > > > > variables
> > > >>> > > > > > > > > > > > > in sql. E.g:
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > CREATE TABLE KafkaTable (
> > > >>> > > > > > > > > > > > > `user_id` BIGINT,
> > > >>> > > > > > > > > > > > > `item_id` BIGINT
> > > >>> > > > > > > > > > > > > ) WITH (
> > > >>> > > > > > > > > > > > > 'connector' = 'kafka',
> > > >>> > > > > > > > > > > > > 'topic' =
> > 'user_behavior',
> > > >>> > > > > > > > > > > > >
> > > >>> 'properties.bootstrap.servers' =
> > > >>> > > > > > > > > > > '{kafka_clusterA}',
> > > >>> > > > > > > > > > > > > 'properties.group.id'
> =
> > > >>> > > > 'testGroup',
> > > >>> > > > > > > > > > > > > 'scan.startup.mode' =
> > > >>> > > > > 'earliest-offset',
> > > >>> > > > > > > > > > > > > 'format' = 'csv'
> > > >>> > > > > > > > > > > > > )
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > Note: 'properties.bootstrap.servers' =
> > > >>> > > > > '{kafka_clusterA}',
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > For variable management, I have some
> > > question:
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > 1. Is the parameter used within the
> team?
> > If
> > > >>> all
> > > >>> > > teams
> > > >>> > > > > > use a
> > > >>> > > > > > > > > > common
> > > >>> > > > > > > > > > > > > parameter, it
> > > >>> > > > > > > > > > > > > needs to be defined for each team,
> right?
> > > >>> > > > > > > > > > > > > 2. Which users can modify variables? Can
> > all
> > > >>> users
> > > >>> > > > > modify
> > > >>> > > > > > > all
> > > >>> > > > > > > > > > > > variables of
> > > >>> > > > > > > > > > > > > their team?
> > > >>> > > > > > > > > > > > > After the variable is modified
> > incorrectly,
> > > >>> many
> > > >>> > > jobs
> > > >>> > > > > may
> > > >>> > > > > > be
> > > >>> > > > > > > > > > > > affected. So I
> > > >>> > > > > > > > > > > > > suggest that
> > > >>> > > > > > > > > > > > > only team admin or system admin have
> > > >>> permission to
> > > >>> > > > > modify,
> > > >>> > > > > > > add
> > > >>> > > > > > > > > and
> > > >>> > > > > > > > > > > > delete
> > > >>> > > > > > > > > > > > > variables.
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > Best
> > > >>> > > > > > > > > > > > > fanrui
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <
> > > >>> > > > > benjobx@gmail.com
> > > >>> > > > > > > >
> > > >>> > > > > > > > > > wrote:
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > hi 阿穆:
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > Thanks for starting a very useful
> > > discussion,
> > > >>> I
> > > >>> > > have a
> > > >>> > > > > > some
> > > >>> > > > > > > > > > > questions:
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > 1. The access boundary problem of
> variable
> > > >>> > > management,
> > > >>> > > > > is
> > > >>> > > > > > > this
> > > >>> > > > > > > > > all
> > > >>> > > > > > > > > > > > > people in the platform can access these
> > > >>> variables(
> > > >>> > > no
> > > >>> > > > > > matter
> > > >>> > > > > > > > who
> > > >>> > > > > > > > > > > > > defines) or that only team members can
> be
> > > >>> access
> > > >>> > > these
> > > >>> > > > > > > > > variables.
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > 2. There are two usage scenarios for
> > > variables
> > > >>> > that
> > > >>> > > I
> > > >>> > > > > know
> > > >>> > > > > > > of,
> > > >>> > > > > > > > > one
> > > >>> > > > > > > > > > > is
> > > >>> > > > > > > > > > > > > in the entry parameter of main, and the
> > > other
> > > >>> is
> > > >>> > in
> > > >>> > > > > > > flinksql.
> > > >>> > > > > > > > > Are
> > > >>> > > > > > > > > > > both
> > > >>> > > > > > > > > > > > > supported, or only supported main entry
> ?
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > about first question, my opinion:
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > 1. The variable management is associated
> > > with
> > > >>> the
> > > >>> > > > team.
> > > >>> > > > > > > After
> > > >>> > > > > > > > > the
> > > >>> > > > > > > > > > > user
> > > >>> > > > > > > > > > > > > logs in to the system, he must select a
> > team
> > > >>> (a
> > > >>> > user
> > > >>> > > > can
> > > >>> > > > > > > > belong
> > > >>> > > > > > > > > to
> > > >>> > > > > > > > > > > one
> > > >>> > > > > > > > > > > > > or more teams), The variable belongs to
> a
> > > team
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > 2. All resources between teams are
> > isolated,
> > > >>> which
> > > >>> > > > means
> > > >>> > > > > > > that
> > > >>> > > > > > > > > > > > > variables are also isolated. In short,
> > > >>> members of
> > > >>> > a
> > > >>> > > > team
> > > >>> > > > > > can
> > > >>> > > > > > > > use
> > > >>> > > > > > > > > > and
> > > >>> > > > > > > > > > > > > define these variables, which are
> > invisible
> > > to
> > > >>> > other
> > > >>> > > > > > > members,
> > > >>> > > > > > > > > and
> > > >>> > > > > > > > > > > > > other members cannot use these
> variables.
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > Welcome to your views
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > Best,
> > > >>> > > > > > > > > > > > > Huajie Wang
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com>
> > > >>> > 于2022年10月7日周五
> > > >>> > > > > > > 14:18写道：
> > > >>> > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > Is it possible to do this variable
> > > >>> support in
> > > >>> > > sql
> > > >>> > > > > as
> > > >>> > > > > > > well
> > > >>> > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > 阿穆 <305343894@qq.com.invalid>
> > > >>> > 于2022年10月7日周五
> > > >>> > > > > > > 13:54写道：
> > > >>> > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > > Hi guys,
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > > I want to start a discussion
> > about
> > > >>> the
> > > >>> > > > > variable
> > > >>> > > > > > > > > > management
> > > >>> > > > > > > > > > > > module. Please
> > > >>> > > > > > > > > > > > > > > see this issue issue-1477.
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > > Background:
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > > In the actual production
> > > >>> environment,
> > > >>> > > Flink
> > > >>> > > > > jobs
> > > >>> > > > > > > are
> > > >>> > > > > > > > > > > > generally complex,
> > > >>> > > > > > > > > > > > > > > and usually require multiple
> > > >>> external
> > > >>> > > > > > components.
> > > >>> > > > > > > > For
> > > >>> > > > > > > > > > > > example, Flink jobs
> > > >>> > > > > > > > > > > > > > > consume data from Kafka, then
> > > >>> connect
> > > >>> > > > external
> > > >>> > > > > > > > > > components
> > > >>> > > > > > > > > > > > such as HBase
> > > >>> > > > > > > > > > > > > > or
> > > >>> > > > > > > > > > > > > > > Redis to obtain additional
> > > business
> > > >>> > > > > information,
> > > >>> > > > > > > and
> > > >>> > > > > > > > > > then
> > > >>> > > > > > > > > > > > write it to the
> > > >>> > > > > > > > > > > > > > > downstream external
> components.
> > > >>> There
> > > >>> > are
> > > >>> > > > the
> > > >>> > > > > > > > > following
> > > >>> > > > > > > > > > > > problems:
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > > 1. The connection information
> of
> > > >>> > external
> > > >>> > > > > > > > components,
> > > >>> > > > > > > > > > such
> > > >>> > > > > > > > > > > > as IP, port
> > > >>> > > > > > > > > > > > > > and
> > > >>> > > > > > > > > > > > > > > user password, needs to be
> > > >>> configured in
> > > >>> > > the
> > > >>> > > > > > > > > application
> > > >>> > > > > > > > > > > > args and
> > > >>> > > > > > > > > > > > > > > transferred to the Flink job,
> so
> > > >>> that
> > > >>> > the
> > > >>> > > > > > > connection
> > > >>> > > > > > > > > > > > information of
> > > >>> > > > > > > > > > > > > > > external components is
> > distributed
> > > >>> in
> > > >>> > > > multiple
> > > >>> > > > > > > > > > > > applications. Once the
> > > >>> > > > > > > > > > > > > > > connection information of
> > external
> > > >>> > > > components
> > > >>> > > > > > > > changes,
> > > >>> > > > > > > > > > > many
> > > >>> > > > > > > > > > > > application
> > > >>> > > > > > > > > > > > > > > args parameters need to be
> > > modified,
> > > >>> > which
> > > >>> > > > > will
> > > >>> > > > > > > lead
> > > >>> > > > > > > > > to
> > > >>> > > > > > > > > > > > high operation
> > > >>> > > > > > > > > > > > > > and
> > > >>> > > > > > > > > > > > > > > maintenance costs.
> > > >>> > > > > > > > > > > > > > > 2. There are many people in
> the
> > > team
> > > >>> > > > > developing
> > > >>> > > > > > > > > > real-time
> > > >>> > > > > > > > > > > > computing jobs.
> > > >>> > > > > > > > > > > > > > > There is no uniform
> > specification
> > > >>> for
> > > >>> > the
> > > >>> > > > > > > connection
> > > >>> > > > > > > > > > > > information of the
> > > >>> > > > > > > > > > > > > > > external components passed to
> > the
> > > >>> job,
> > > >>> > > > > resulting
> > > >>> > > > > > > in
> > > >>> > > > > > > > > > > > different parameter
> > > >>> > > > > > > > > > > > > > > names of the same component.
> > This
> > > is
> > > >>> > > > difficult
> > > >>> > > > > > to
> > > >>> > > > > > > > > count
> > > >>> > > > > > > > > > > > which external
> > > >>> > > > > > > > > > > > > > > components are dependent.
> > > >>> > > > > > > > > > > > > > > 3. In production practice,
> there
> > > are
> > > >>> > > usually
> > > >>> > > > > > > > multiple
> > > >>> > > > > > > > > > sets
> > > >>> > > > > > > > > > > > of
> > > >>> > > > > > > > > > > > > > > environments, such as test
> > > >>> environment
> > > >>> > and
> > > >>> > > > > > > > production
> > > >>> > > > > > > > > > > > environment. It is
> > > >>> > > > > > > > > > > > > > > not intuitive to judge
> whether a
> > > >>> > component
> > > >>> > > > > > belongs
> > > >>> > > > > > > > to
> > > >>> > > > > > > > > a
> > > >>> > > > > > > > > > > > test environment
> > > >>> > > > > > > > > > > > > > or
> > > >>> > > > > > > > > > > > > > > a production environment
> simply
> > > >>> through
> > > >>> > IP
> > > >>> > > > and
> > > >>> > > > > > > > ports.
> > > >>> > > > > > > > > > Even
> > > >>> > > > > > > > > > > > if it can be
> > > >>> > > > > > > > > > > > > > > judged, there are some
> > omissions.
> > > In
> > > >>> > this
> > > >>> > > > way,
> > > >>> > > > > > the
> > > >>> > > > > > > > > > > > connection information
> > > >>> > > > > > > > > > > > > > > online to the production
> > > >>> environment may
> > > >>> > > be
> > > >>> > > > > > > external
> > > >>> > > > > > > > > > > > components of the
> > > >>> > > > > > > > > > > > > > test
> > > >>> > > > > > > > > > > > > > > environment, or components of
> > the
> > > >>> > > production
> > > >>> > > > > > > > > environment
> > > >>> > > > > > > > > > > are
> > > >>> > > > > > > > > > > > > > inadvertently
> > > >>> > > > > > > > > > > > > > > configured during testing,
> > leading
> > > >>> to
> > > >>> > > > > production
> > > >>> > > > > > > > > > failures.
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > > Proposal:
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > > 1. Create a unified module to
> > > manage
> > > >>> > > > > variables.
> > > >>> > > > > > > > Users
> > > >>> > > > > > > > > > can
> > > >>> > > > > > > > > > > > define their
> > > >>> > > > > > > > > > > > > > own
> > > >>> > > > > > > > > > > > > > > variables in this module.
> > Variable
> > > >>> > > > information
> > > >>> > > > > > > > > includes
> > > >>> > > > > > > > > > > > variable code
> > > >>> > > > > > > > > > > > > > > (passed to Flink program as
> > > >>> parameter
> > > >>> > > name),
> > > >>> > > > > > > > variable
> > > >>> > > > > > > > > > > > value, variable
> > > >>> > > > > > > > > > > > > name,
> > > >>> > > > > > > > > > > > > > > variable description, and
> which
> > > >>> > > applications
> > > >>> > > > > > > depend
> > > >>> > > > > > > > on
> > > >>> > > > > > > > > > > > > > > 2. In the same Team module,
> the
> > > >>> variable
> > > >>> > > > code
> > > >>> > > > > or
> > > >>> > > > > > > > > > variable
> > > >>> > > > > > > > > > > > name is unique
> > > >>> > > > > > > > > > > > > > > - By using placeholders, such
> as
> > > >>> adding
> > > >>> > --
> > > >>> > > > > kafka
> > > >>> > > > > > > > > > {mykafka}
> > > >>> > > > > > > > > > > > to the args of
> > > >>> > > > > > > > > > > > > > > the application, mykafka is
> the
> > > >>> variable
> > > >>> > > > code
> > > >>> > > > > > of a
> > > >>> > > > > > > > > > > variable
> > > >>> > > > > > > > > > > > > > > - Search by variable code or
> > > >>> variable
> > > >>> > > name.
> > > >>> > > > > > After
> > > >>> > > > > > > a
> > > >>> > > > > > > > > > > > variable is selected,
> > > >>> > > > > > > > > > > > > > > the parameter transfer method
> is
> > > >>> > displayed
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > > If you have any suggestions,
> > > >>> welcome to
> > > >>> > > > > > > participate
> > > >>> > > > > > > > in
> > > >>> > > > > > > > > > the
> > > >>> > > > > > > > > > > > discussion on
> > > >>> > > > > > > > > > > > > > > the mail list or issue, I look
> > > >>> forward
> > > >>> > to
> > > >>> > > > your
> > > >>> > > > > > > > > feedback.
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > >
> > > >>> > > > > > > > > > > > > > > Best wishes
> > > >>> > > > > > > > > > > > > >
> > > >>> > > > > > > > > > >
> > > >>> > > > > > > > > >
> > > >>> > > > > > > > >
> > > >>> > > > > > > >
> > > >>> > > > > > >
> > > >>> > > > > >
> > > >>> > > > >
> > > >>> > > >
> > > >>> > >
> > > >>> >
> > > >>>
> > > >>
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Chunjin Mu <ma...@gmail.com>.
Hi fanru,
> I have a little question. If I copy some sql from a old app, and the sql
includes some placeholders. These placeholders will take effect in the
new app, right?
yes, the copied placeholder will take effect if the variable exists.
I have added all the information to proposal issue-1477[1]，include the
questions you asked above， if there is more discussion later, I will add
this to the proposal.
[1] https://github.com/apache/incubator-streampark/issues/1477
Rui Fan <19...@gmail.com> 于2022年10月15日周六 13:17写道：
> Hi chunjin,
>
> Thanks for your detailed information.
>
> > When the focus is lost, it is judged whether there is a variable
> placeholder, and if so, a verification button is displayed to prompt the
> user to perform verification. When the user clicks the verification button,
> the replaced content of the placeholder is displayed, and only the user
> clicks the verification button to click the submit button
>
> I have a little question. If I copy some sql from a old app, and the sql
> includes some placeholders. These placeholders will take effect in the
> new app, right?
>
> BTW, could you add all information to your proposal issue-1477[1]?
> The mail list is for discussion only, and we should maintain the final
> solution in the Proposal.
>
>
> [1] https://github.com/apache/incubator-streampark/issues/1477
>
> Best
> fanrui
>
> On Sat, Oct 15, 2022 at 11:56 AM Chunjin Mu <ma...@gmail.com> wrote:
>
> > Hi guys,
> >
> > Thanks for the discussion and great suggestions.
> >
> > Here I summarize the functions to be completed in version 1.2.4
> >
> > 1.The table structure of the variable contains the following main
> contents
> > - variable code is placeholder, For example the variable code is
> > "collect.kafka.cluster", it will be referenced as
> ${collect.kafka.cluster},
> > it is unique within a team
> > - variable name is removed, after discussion, it is not necessarily
> > retained
> > - variable value is the value of a specific variable, such as ip and port
> > - description is a detailed introduction
> > - team is the team the variable belongs to
> >
> > 2.Basic functions of variable management
> > - Add and modify variables: need to check whether the variable code is
> > unique under the current team
> > - Deleting a variable: It is necessary to check whether it has been
> > referenced by the Application under the team to which the variable
> belongs.
> > If there is metadata management in the future, it is also necessary to
> > check whether it has been referenced by the metadata. If it is
> referenced,
> > it will prompt that the variable is referenced and cannot be deleted.
> >
> > 3.References in Flink sql and program args
> > - In the application, the variables are uniformly referenced in the form
> of
> > placeholders ${xxx}, such as in Flink sql, program args
> > - For a better experience, it is necessary to provide the function of
> > following the search prompt when entering variable placeholders. For
> > example, when entering "${", the variable list is displayed, and when
> > "${kafka" is entered, the display contains variables for kafka strings,
> in
> > "variable code" or "description".
> > - When the focus is lost, it is judged whether there is a variable
> > placeholder, and if so, a verification button is displayed to prompt the
> > user to perform verification. When the user clicks the verification
> button,
> > the replaced content of the placeholder is displayed, and only the user
> > clicks the verification button to click the submit button
> > - If the user enters "okir23${j2Rts}pEt", this placeholder "${j2Rts}"
> will
> > not be replaced if "j2Rts" is not a variable we defined
> >
> > 4.Future plan
> > - Display the application or metadata list of a referenced variable in
> the
> > variable basic management module
> > - Improve user experience
> >
> > Best,
> > Chunjin Mu
> >
> > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:28写道：
> >
> > > When we have dozens or dozens of variables, this phenomenon may be more
> > > prominent
> > >
> > >
> > > Best,
> > > Chunjin Mu
> > >
> > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:26写道：
> > >
> > >> When a person asks which kafka should be used, I tell him in Chinese
> > that
> > >> he should use "采集整合的kafka", in fact the variable code is
> collect.kafka,
> > but
> > >> he is searching for gather keyword, it is difficult to just pass
> > >> collect.kafka The intuitive understanding is to "采集整合的kafka", because
> > >> the information contained in the variable code is still not that much.
> > >>
> > >> Huajie Wang <be...@gmail.com> 于2022年10月14日周五 22:17写道：
> > >>
> > >>> I just thought about it, there is really no need for this name.
> > >>>
> > >>>
> > >>>
> > >>> Best,
> > >>> Huajie Wang
> > >>>
> > >>>
> > >>>
> > >>> Rui Fan <19...@gmail.com> 于2022年10月14日周五 21:54写道：
> > >>>
> > >>> > Hi guys,
> > >>> >
> > >>> > I understand chunjin's solution is this:
> > >>> >
> > >>> > - variable code is placeholder
> > >>> > - variable name is an introduction
> > >>> > - Description is a detailed introduction
> > >>> >
> > >>> > I think introduction and detail are redundant.
> > >>> >
> > >>> > For example: there are two kafka clusters, the Beijing kafka
> > >>> > cluster and the Hangzhou kafka cluster.
> > >>> >
> > >>> > If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why
> > >>> > does the introduction still need `北京 kafka 集群` and
> > >>> > `杭州 kafka 集群`? placeholder is already clear.
> > >>> >
> > >>> > If dev may forget the placeholder, then dev may also
> > >>> > forget the introduction.
> > >>> >
> > >>> > Of course, if the placeholder is kafka1 kafka2, an
> > >>> > introduction is required. Why do we need an introduction
> > >>> > at this point, because the placeholder name is unreasonable.
> > >>> >
> > >>> > Just like variable naming in Java code, there is no need to
> > >>> > introduce an introduction with a clear name. If the meaning of
> > >>> > the variable is complicated, we also need to write a detailed
> > >>> > description.
> > >>> >
> > >>> > Finally, I want to say: less is more. If we want to make
> > >>> > something complicated, please give everyone a strong reason.
> > >>> >
> > >>> > Best
> > >>> > fanrui
> > >>> >
> > >>> > On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <be...@gmail.com>
> > wrote:
> > >>> >
> > >>> > > Hi guys,
> > >>> > >
> > >>> > > I agree with Chunjin, I think variable name is useful, Chunjin
> also
> > >>> > > explained the reason.
> > >>> > >
> > >>> > >
> > >>> > > Best,
> > >>> > > Huajie Wang
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 20:22写道：
> > >>> > >
> > >>> > > > Hi, Rui Fan
> > >>> > > > I'll start a discussion later.
> > >>> > > >
> > >>> > > > Best
> > >>> > > > Chunjin Mu
> > >>> > > >
> > >>> > > > Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
> > >>> > > >
> > >>> > > > > Hi guys,
> > >>> > > > >
> > >>> > > > > I agree with monster. And I suggest use variable name,
> variable
> > >>> value
> > >>> > > and
> > >>> > > > > description.
> > >>> > > > >
> > >>> > > > > By the way, there are some problems still need to confirm.
> > >>> What’s the
> > >>> > > > > placeholder? And how to use them when edit sql or datastream
> > job?
> > >>> > It’s
> > >>> > > > > better to discuss these asap.
> > >>> > > > >
> > >>> > > > > Best
> > >>> > > > > fanrui
> > >>> > > > >
> > >>> > > > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <
> macksonmu0@gmail.com
> > >
> > >>> > wrote:
> > >>> > > > >
> > >>> > > > > > hi, huajie, fan rui, monster
> > >>> > > > > >
> > >>> > > > > > Thank you for your suggestions.
> > >>> > > > > >
> > >>> > > > > > Now please Huajie to confirm how to do it. If remove the
> > >>> variable
> > >>> > > > name, I
> > >>> > > > > > can also accept it here. I will do small secondary
> > development
> > >>> on
> > >>> > our
> > >>> > > > > > production system based on the community to meet internal
> > >>> needs.
> > >>> > > > > >
> > >>> > > > > > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五 15:58写道：
> > >>> > > > > >
> > >>> > > > > > > "外部采集整合kafka" is the Chinese description of variable, so
> it
> > >>> makes
> > >>> > > > sense
> > >>> > > > > > to
> > >>> > > > > > > put it under `Description`.
> > >>> > > > > > >
> > >>> > > > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五
> 12:01写道：
> > >>> > > > > > >
> > >>> > > > > > > > The variable management module has three properties,
> > >>> including
> > >>> > > > > variable
> > >>> > > > > > > > code, variable name, and description, For example, the
> > >>> variable
> > >>> > > > code
> > >>> > > > > is
> > >>> > > > > > > > "external.kafka.collect.brokers", and the variable name
> > is
> > >>> > > > > > "外部采集整合kafka".
> > >>> > > > > > > > When we created the application, I forgot what the
> > variable
> > >>> > code
> > >>> > > > was,
> > >>> > > > > > > but I
> > >>> > > > > > > > knew it was called "采集整合", so I can search Enter the
> > "采集整合"
> > >>> > > > keywords
> > >>> > > > > in
> > >>> > > > > > > the
> > >>> > > > > > > > box, and of course you can also search for kafka
> > keywords.
> > >>> > > > > > > >
> > >>> > > > > > > > At the same time, I want to know more detailed
> > information
> > >>> > about
> > >>> > > > this
> > >>> > > > > > > > kafka, such as who the operator is, what is the data
> > >>> retention
> > >>> > > > > period,
> > >>> > > > > > > and
> > >>> > > > > > > > how big is the cluster size, but I can't use the
> > >>> description to
> > >>> > > > > search,
> > >>> > > > > > > so
> > >>> > > > > > > > I can only search or variable code or variable name,
> and
> > >>> the
> > >>> > > > variable
> > >>> > > > > > > name
> > >>> > > > > > > > is also for a simple display.
> > >>> > > > > > > >
> > >>> > > > > > > > Although external.kafka.collect.brokers already
> contains
> > a
> > >>> lot
> > >>> > of
> > >>> > > > > > > > information, I think this is just a naming convention.
> > >>> When we
> > >>> > > > > discuss
> > >>> > > > > > > this
> > >>> > > > > > > > variable, we will discuss it by using the "采集整合 kafka"
> > >>> > > > > > > >
> > >>> > > > > > > > We can discuss whether there is a problem with this
> > design,
> > >>> > what
> > >>> > > do
> > >>> > > > > you
> > >>> > > > > > > > think?
> > >>> > > > > > > >
> > >>> > > > > > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
> > >>> > > > > > > >
> > >>> > > > > > > > > Hi huajie:
> > >>> > > > > > > > >
> > >>> > > > > > > > > > If it is manually entered, it will always be
> treated
> > as
> > >>> > > > ordinary
> > >>> > > > > > > text.
> > >>> > > > > > > > >
> > >>> > > > > > > > > Users always copy some code from an old job to a new
> > job.
> > >>> > > > > > > > > I'm afraid it might not meet the user's expectations.
> > >>> > > > > > > > >
> > >>> > > > > > > > > Best,
> > >>> > > > > > > > > fanrui
> > >>> > > > > > > > >
> > >>> > > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <
> > >>> > benjobx@gmail.com>
> > >>> > > > > > wrote:
> > >>> > > > > > > > >
> > >>> > > > > > > > > > hi 田向阳:
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > The current discussion is about variable
> management,
> > >>> which
> > >>> > > is a
> > >>> > > > > > > > > relatively
> > >>> > > > > > > > > > independent function, No other modules involved,
> you
> > >>> can
> > >>> > > check
> > >>> > > > > the
> > >>> > > > > > > > thread
> > >>> > > > > > > > > > [1]
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > [1] thread:
> > >>> > > > > > > > > >
> > >>> > > >
> https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > Best,
> > >>> > > > > > > > > > Huajie Wang
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > >>> > > > > > > > > >
> > >>> > > > > > > > > > > is this module include a resource management
> cente
> > r
> > >>> > where
> > >>> > > > jars
> > >>> > > > > > can
> > >>> > > > > > > > be
> > >>> > > > > > > > > > > chosen to provide to create a job
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > ---- 回复的原邮件 ----
> > >>> > > > > > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > >>> > > > > > > > > > > | 日期 | 2022年10月08日 12:07 |
> > >>> > > > > > > > > > > | 收件人 | dev@streampark.apache.org<
> > >>> > > dev@streampark.apache.org>
> > >>> > > > |
> > >>> > > > > > > > > > > | 抄送至 | |
> > >>> > > > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable
> > >>> management
> > >>> > > > > module |
> > >>> > > > > > > > > > > hi 阿穆:
> > >>> > > > > > > > > > > Thank you for the clarification, My question is
> > >>> answered,
> > >>> > > but
> > >>> > > > > > now I
> > >>> > > > > > > > > have
> > >>> > > > > > > > > > > one more concern,
> > >>> > > > > > > > > > > I'm currently using {var} for variables, and I'm
> > >>> worried
> > >>> > > that
> > >>> > > > > one
> > >>> > > > > > > of
> > >>> > > > > > > > > the
> > >>> > > > > > > > > > > user's parameters includes "{" and "}", e.g :
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > --password 'mypass{123}abc'
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > So how do we parse parse this? How about ${var}
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > Best,
> > >>> > > > > > > > > > > Huajie Wang
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六
> > 11:36写道：
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > Thanks for your reply.
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > I have integrated your replies. I really don't
> > pay
> > >>> much
> > >>> > > > > > attention
> > >>> > > > > > > > to
> > >>> > > > > > > > > > SQL,
> > >>> > > > > > > > > > > > so I will do the following.
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > 1.Each team has independent variables that do
> not
> > >>> > affect
> > >>> > > > each
> > >>> > > > > > > other
> > >>> > > > > > > > > and
> > >>> > > > > > > > > > > > are isolated from each other, but the same
> > >>> variable is
> > >>> > > > unique
> > >>> > > > > > > under
> > >>> > > > > > > > > the
> > >>> > > > > > > > > > > > same team, so you need to specify the team when
> > >>> > creating
> > >>> > > > > > > variables
> > >>> > > > > > > > > > (users
> > >>> > > > > > > > > > > > may belong to multiple teams, and the teams of
> > the
> > >>> > > current
> > >>> > > > > user
> > >>> > > > > > > > will
> > >>> > > > > > > > > be
> > >>> > > > > > > > > > > > displayed here for users to choose)
> > >>> > > > > > > > > > > > 2.For Flink jar jobs, there are two ways to
> pass
> > >>> > > variables
> > >>> > > > > > > > > > > > - By using placeholders, such as adding --kafka
> > >>> > {mykafka}
> > >>> > > > to
> > >>> > > > > > the
> > >>> > > > > > > > args
> > >>> > > > > > > > > > of
> > >>> > > > > > > > > > > > the application, 'mykafka' is the variable code
> > of
> > >>> a
> > >>> > > > variable
> > >>> > > > > > > > > > > > - Search by variable code or variable name.
> > After a
> > >>> > > > variable
> > >>> > > > > is
> > >>> > > > > > > > > > selected,
> > >>> > > > > > > > > > > > the parameter transfer method is displayed
> > >>> > > > > > > > > > > > 3.Flink sql will use variables as placeholders,
> > >>> E.g:
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > CREATE TABLE KafkaTableA (
> > >>> > > > > > > > > > > > `user_id` BIGINT,
> > >>> > > > > > > > > > > > `item_id` BIGINT,
> > >>> > > > > > > > > > > > `type` STRING
> > >>> > > > > > > > > > > > ) WITH (
> > >>> > > > > > > > > > > > 'connector' = 'kafka',
> > >>> > > > > > > > > > > > 'topic' = 'user_behaviorA',
> > >>> > > > > > > > > > > > 'properties.bootstrap.servers' =
> > >>> > > > > > > > > '{kafka_clusterA}',
> > >>> > > > > > > > > > > --
> > >>> > > > > > > > > > > > 'kafka_ ClusterA' is the variable code
> > >>> > > > > > > > > > > > 'properties.group.id' =
> > 'testGroupA',
> > >>> > > > > > > > > > > > 'scan.startup.mode' =
> > >>> 'earliest-offset',
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > CREATE TABLE KafkaTableB (
> > >>> > > > > > > > > > > > `user_id` BIGINT,
> > >>> > > > > > > > > > > > `item_id` BIGINT
> > >>> > > > > > > > > > > > ) WITH (
> > >>> > > > > > > > > > > > 'connector' = 'kafka',
> > >>> > > > > > > > > > > > 'topic' = 'user_behaviorB',
> > >>> > > > > > > > > > > > 'properties.bootstrap.servers' =
> > >>> > > > > > > > > '{kafka_clusterB}',
> > >>> > > > > > > > > > > --
> > >>> > > > > > > > > > > > 'kafka_ ClusterB' is the variable code
> > >>> > > > > > > > > > > > 'properties.group.id' =
> > 'testGroupB',
> > >>> > > > > > > > > > > > 'scan.startup.mode' =
> > >>> 'earliest-offset',
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id,
> item_id,
> > >>> > '{type}'
> > >>> > > > > FROM
> > >>> > > > > > > > > > > > KafkaTableB -- 'type' is the
> variable
> > >>> code
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > 4.In the future, we can consider setting some
> > >>> system
> > >>> > > > built-in
> > >>> > > > > > > > > variables
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > Best,
> > >>> > > > > > > > > > > > ChunJin Mu
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
> > >>> > > > 1996fanrui@gmail.com
> > >>> > > > > > >
> > >>> > > > > > > > > > wrote:
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > > Hi 阿穆:
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > It's a great proposal.
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > For usage scenarios, I strongly recommend
> > >>> support
> > >>> > > for
> > >>> > > > > > > > replacing
> > >>> > > > > > > > > > > > variables
> > >>> > > > > > > > > > > > > in sql. E.g:
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > CREATE TABLE KafkaTable (
> > >>> > > > > > > > > > > > > `user_id` BIGINT,
> > >>> > > > > > > > > > > > > `item_id` BIGINT
> > >>> > > > > > > > > > > > > ) WITH (
> > >>> > > > > > > > > > > > > 'connector' = 'kafka',
> > >>> > > > > > > > > > > > > 'topic' =
> 'user_behavior',
> > >>> > > > > > > > > > > > >
> > >>> 'properties.bootstrap.servers' =
> > >>> > > > > > > > > > > '{kafka_clusterA}',
> > >>> > > > > > > > > > > > > 'properties.group.id' =
> > >>> > > > 'testGroup',
> > >>> > > > > > > > > > > > > 'scan.startup.mode' =
> > >>> > > > > 'earliest-offset',
> > >>> > > > > > > > > > > > > 'format' = 'csv'
> > >>> > > > > > > > > > > > > )
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > Note: 'properties.bootstrap.servers' =
> > >>> > > > > '{kafka_clusterA}',
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > For variable management, I have some
> > question:
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > 1. Is the parameter used within the team?
> If
> > >>> all
> > >>> > > teams
> > >>> > > > > > use a
> > >>> > > > > > > > > > common
> > >>> > > > > > > > > > > > > parameter, it
> > >>> > > > > > > > > > > > > needs to be defined for each team, right?
> > >>> > > > > > > > > > > > > 2. Which users can modify variables? Can
> all
> > >>> users
> > >>> > > > > modify
> > >>> > > > > > > all
> > >>> > > > > > > > > > > > variables of
> > >>> > > > > > > > > > > > > their team?
> > >>> > > > > > > > > > > > > After the variable is modified
> incorrectly,
> > >>> many
> > >>> > > jobs
> > >>> > > > > may
> > >>> > > > > > be
> > >>> > > > > > > > > > > > affected. So I
> > >>> > > > > > > > > > > > > suggest that
> > >>> > > > > > > > > > > > > only team admin or system admin have
> > >>> permission to
> > >>> > > > > modify,
> > >>> > > > > > > add
> > >>> > > > > > > > > and
> > >>> > > > > > > > > > > > delete
> > >>> > > > > > > > > > > > > variables.
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > Best
> > >>> > > > > > > > > > > > > fanrui
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <
> > >>> > > > > benjobx@gmail.com
> > >>> > > > > > > >
> > >>> > > > > > > > > > wrote:
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > > hi 阿穆:
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > Thanks for starting a very useful
> > discussion,
> > >>> I
> > >>> > > have a
> > >>> > > > > > some
> > >>> > > > > > > > > > > questions:
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > 1. The access boundary problem of variable
> > >>> > > management,
> > >>> > > > > is
> > >>> > > > > > > this
> > >>> > > > > > > > > all
> > >>> > > > > > > > > > > > > people in the platform can access these
> > >>> variables(
> > >>> > > no
> > >>> > > > > > matter
> > >>> > > > > > > > who
> > >>> > > > > > > > > > > > > defines) or that only team members can be
> > >>> access
> > >>> > > these
> > >>> > > > > > > > > variables.
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > 2. There are two usage scenarios for
> > variables
> > >>> > that
> > >>> > > I
> > >>> > > > > know
> > >>> > > > > > > of,
> > >>> > > > > > > > > one
> > >>> > > > > > > > > > > is
> > >>> > > > > > > > > > > > > in the entry parameter of main, and the
> > other
> > >>> is
> > >>> > in
> > >>> > > > > > > flinksql.
> > >>> > > > > > > > > Are
> > >>> > > > > > > > > > > both
> > >>> > > > > > > > > > > > > supported, or only supported main entry ?
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > about first question, my opinion:
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > 1. The variable management is associated
> > with
> > >>> the
> > >>> > > > team.
> > >>> > > > > > > After
> > >>> > > > > > > > > the
> > >>> > > > > > > > > > > user
> > >>> > > > > > > > > > > > > logs in to the system, he must select a
> team
> > >>> (a
> > >>> > user
> > >>> > > > can
> > >>> > > > > > > > belong
> > >>> > > > > > > > > to
> > >>> > > > > > > > > > > one
> > >>> > > > > > > > > > > > > or more teams), The variable belongs to a
> > team
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > 2. All resources between teams are
> isolated,
> > >>> which
> > >>> > > > means
> > >>> > > > > > > that
> > >>> > > > > > > > > > > > > variables are also isolated. In short,
> > >>> members of
> > >>> > a
> > >>> > > > team
> > >>> > > > > > can
> > >>> > > > > > > > use
> > >>> > > > > > > > > > and
> > >>> > > > > > > > > > > > > define these variables, which are
> invisible
> > to
> > >>> > other
> > >>> > > > > > > members,
> > >>> > > > > > > > > and
> > >>> > > > > > > > > > > > > other members cannot use these variables.
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > Welcome to your views
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > Best,
> > >>> > > > > > > > > > > > > Huajie Wang
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com>
> > >>> > 于2022年10月7日周五
> > >>> > > > > > > 14:18写道：
> > >>> > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > Is it possible to do this variable
> > >>> support in
> > >>> > > sql
> > >>> > > > > as
> > >>> > > > > > > well
> > >>> > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > 阿穆 <305343894@qq.com.invalid>
> > >>> > 于2022年10月7日周五
> > >>> > > > > > > 13:54写道：
> > >>> > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > Hi guys,
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > I want to start a discussion
> about
> > >>> the
> > >>> > > > > variable
> > >>> > > > > > > > > > management
> > >>> > > > > > > > > > > > module. Please
> > >>> > > > > > > > > > > > > > > see this issue issue-1477.
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > Background:
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > In the actual production
> > >>> environment,
> > >>> > > Flink
> > >>> > > > > jobs
> > >>> > > > > > > are
> > >>> > > > > > > > > > > > generally complex,
> > >>> > > > > > > > > > > > > > > and usually require multiple
> > >>> external
> > >>> > > > > > components.
> > >>> > > > > > > > For
> > >>> > > > > > > > > > > > example, Flink jobs
> > >>> > > > > > > > > > > > > > > consume data from Kafka, then
> > >>> connect
> > >>> > > > external
> > >>> > > > > > > > > > components
> > >>> > > > > > > > > > > > such as HBase
> > >>> > > > > > > > > > > > > > or
> > >>> > > > > > > > > > > > > > > Redis to obtain additional
> > business
> > >>> > > > > information,
> > >>> > > > > > > and
> > >>> > > > > > > > > > then
> > >>> > > > > > > > > > > > write it to the
> > >>> > > > > > > > > > > > > > > downstream external components.
> > >>> There
> > >>> > are
> > >>> > > > the
> > >>> > > > > > > > > following
> > >>> > > > > > > > > > > > problems:
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > 1. The connection information of
> > >>> > external
> > >>> > > > > > > > components,
> > >>> > > > > > > > > > such
> > >>> > > > > > > > > > > > as IP, port
> > >>> > > > > > > > > > > > > > and
> > >>> > > > > > > > > > > > > > > user password, needs to be
> > >>> configured in
> > >>> > > the
> > >>> > > > > > > > > application
> > >>> > > > > > > > > > > > args and
> > >>> > > > > > > > > > > > > > > transferred to the Flink job, so
> > >>> that
> > >>> > the
> > >>> > > > > > > connection
> > >>> > > > > > > > > > > > information of
> > >>> > > > > > > > > > > > > > > external components is
> distributed
> > >>> in
> > >>> > > > multiple
> > >>> > > > > > > > > > > > applications. Once the
> > >>> > > > > > > > > > > > > > > connection information of
> external
> > >>> > > > components
> > >>> > > > > > > > changes,
> > >>> > > > > > > > > > > many
> > >>> > > > > > > > > > > > application
> > >>> > > > > > > > > > > > > > > args parameters need to be
> > modified,
> > >>> > which
> > >>> > > > > will
> > >>> > > > > > > lead
> > >>> > > > > > > > > to
> > >>> > > > > > > > > > > > high operation
> > >>> > > > > > > > > > > > > > and
> > >>> > > > > > > > > > > > > > > maintenance costs.
> > >>> > > > > > > > > > > > > > > 2. There are many people in the
> > team
> > >>> > > > > developing
> > >>> > > > > > > > > > real-time
> > >>> > > > > > > > > > > > computing jobs.
> > >>> > > > > > > > > > > > > > > There is no uniform
> specification
> > >>> for
> > >>> > the
> > >>> > > > > > > connection
> > >>> > > > > > > > > > > > information of the
> > >>> > > > > > > > > > > > > > > external components passed to
> the
> > >>> job,
> > >>> > > > > resulting
> > >>> > > > > > > in
> > >>> > > > > > > > > > > > different parameter
> > >>> > > > > > > > > > > > > > > names of the same component.
> This
> > is
> > >>> > > > difficult
> > >>> > > > > > to
> > >>> > > > > > > > > count
> > >>> > > > > > > > > > > > which external
> > >>> > > > > > > > > > > > > > > components are dependent.
> > >>> > > > > > > > > > > > > > > 3. In production practice, there
> > are
> > >>> > > usually
> > >>> > > > > > > > multiple
> > >>> > > > > > > > > > sets
> > >>> > > > > > > > > > > > of
> > >>> > > > > > > > > > > > > > > environments, such as test
> > >>> environment
> > >>> > and
> > >>> > > > > > > > production
> > >>> > > > > > > > > > > > environment. It is
> > >>> > > > > > > > > > > > > > > not intuitive to judge whether a
> > >>> > component
> > >>> > > > > > belongs
> > >>> > > > > > > > to
> > >>> > > > > > > > > a
> > >>> > > > > > > > > > > > test environment
> > >>> > > > > > > > > > > > > > or
> > >>> > > > > > > > > > > > > > > a production environment simply
> > >>> through
> > >>> > IP
> > >>> > > > and
> > >>> > > > > > > > ports.
> > >>> > > > > > > > > > Even
> > >>> > > > > > > > > > > > if it can be
> > >>> > > > > > > > > > > > > > > judged, there are some
> omissions.
> > In
> > >>> > this
> > >>> > > > way,
> > >>> > > > > > the
> > >>> > > > > > > > > > > > connection information
> > >>> > > > > > > > > > > > > > > online to the production
> > >>> environment may
> > >>> > > be
> > >>> > > > > > > external
> > >>> > > > > > > > > > > > components of the
> > >>> > > > > > > > > > > > > > test
> > >>> > > > > > > > > > > > > > > environment, or components of
> the
> > >>> > > production
> > >>> > > > > > > > > environment
> > >>> > > > > > > > > > > are
> > >>> > > > > > > > > > > > > > inadvertently
> > >>> > > > > > > > > > > > > > > configured during testing,
> leading
> > >>> to
> > >>> > > > > production
> > >>> > > > > > > > > > failures.
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > Proposal:
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > 1. Create a unified module to
> > manage
> > >>> > > > > variables.
> > >>> > > > > > > > Users
> > >>> > > > > > > > > > can
> > >>> > > > > > > > > > > > define their
> > >>> > > > > > > > > > > > > > own
> > >>> > > > > > > > > > > > > > > variables in this module.
> Variable
> > >>> > > > information
> > >>> > > > > > > > > includes
> > >>> > > > > > > > > > > > variable code
> > >>> > > > > > > > > > > > > > > (passed to Flink program as
> > >>> parameter
> > >>> > > name),
> > >>> > > > > > > > variable
> > >>> > > > > > > > > > > > value, variable
> > >>> > > > > > > > > > > > > name,
> > >>> > > > > > > > > > > > > > > variable description, and which
> > >>> > > applications
> > >>> > > > > > > depend
> > >>> > > > > > > > on
> > >>> > > > > > > > > > > > > > > 2. In the same Team module, the
> > >>> variable
> > >>> > > > code
> > >>> > > > > or
> > >>> > > > > > > > > > variable
> > >>> > > > > > > > > > > > name is unique
> > >>> > > > > > > > > > > > > > > - By using placeholders, such as
> > >>> adding
> > >>> > --
> > >>> > > > > kafka
> > >>> > > > > > > > > > {mykafka}
> > >>> > > > > > > > > > > > to the args of
> > >>> > > > > > > > > > > > > > > the application, mykafka is the
> > >>> variable
> > >>> > > > code
> > >>> > > > > > of a
> > >>> > > > > > > > > > > variable
> > >>> > > > > > > > > > > > > > > - Search by variable code or
> > >>> variable
> > >>> > > name.
> > >>> > > > > > After
> > >>> > > > > > > a
> > >>> > > > > > > > > > > > variable is selected,
> > >>> > > > > > > > > > > > > > > the parameter transfer method is
> > >>> > displayed
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > If you have any suggestions,
> > >>> welcome to
> > >>> > > > > > > participate
> > >>> > > > > > > > in
> > >>> > > > > > > > > > the
> > >>> > > > > > > > > > > > discussion on
> > >>> > > > > > > > > > > > > > > the mail list or issue, I look
> > >>> forward
> > >>> > to
> > >>> > > > your
> > >>> > > > > > > > > feedback.
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > >
> > >>> > > > > > > > > > > > > > > Best wishes
> > >>> > > > > > > > > > > > > >
> > >>> > > > > > > > > > >
> > >>> > > > > > > > > >
> > >>> > > > > > > > >
> > >>> > > > > > > >
> > >>> > > > > > >
> > >>> > > > > >
> > >>> > > > >
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Rui Fan <19...@gmail.com>.
Hi chunjin,
Thanks for your detailed information.
> When the focus is lost, it is judged whether there is a variable
placeholder, and if so, a verification button is displayed to prompt the
user to perform verification. When the user clicks the verification button,
the replaced content of the placeholder is displayed, and only the user
clicks the verification button to click the submit button
I have a little question. If I copy some sql from a old app, and the sql
includes some placeholders. These placeholders will take effect in the
new app, right?
BTW, could you add all information to your proposal issue-1477[1]?
The mail list is for discussion only, and we should maintain the final
solution in the Proposal.
[1] https://github.com/apache/incubator-streampark/issues/1477
Best
fanrui
On Sat, Oct 15, 2022 at 11:56 AM Chunjin Mu <ma...@gmail.com> wrote:
> Hi guys,
>
> Thanks for the discussion and great suggestions.
>
> Here I summarize the functions to be completed in version 1.2.4
>
> 1.The table structure of the variable contains the following main contents
> - variable code is placeholder, For example the variable code is
> "collect.kafka.cluster", it will be referenced as ${collect.kafka.cluster},
> it is unique within a team
> - variable name is removed, after discussion, it is not necessarily
> retained
> - variable value is the value of a specific variable, such as ip and port
> - description is a detailed introduction
> - team is the team the variable belongs to
>
> 2.Basic functions of variable management
> - Add and modify variables: need to check whether the variable code is
> unique under the current team
> - Deleting a variable: It is necessary to check whether it has been
> referenced by the Application under the team to which the variable belongs.
> If there is metadata management in the future, it is also necessary to
> check whether it has been referenced by the metadata. If it is referenced,
> it will prompt that the variable is referenced and cannot be deleted.
>
> 3.References in Flink sql and program args
> - In the application, the variables are uniformly referenced in the form of
> placeholders ${xxx}, such as in Flink sql, program args
> - For a better experience, it is necessary to provide the function of
> following the search prompt when entering variable placeholders. For
> example, when entering "${", the variable list is displayed, and when
> "${kafka" is entered, the display contains variables for kafka strings, in
> "variable code" or "description".
> - When the focus is lost, it is judged whether there is a variable
> placeholder, and if so, a verification button is displayed to prompt the
> user to perform verification. When the user clicks the verification button,
> the replaced content of the placeholder is displayed, and only the user
> clicks the verification button to click the submit button
> - If the user enters "okir23${j2Rts}pEt", this placeholder "${j2Rts}" will
> not be replaced if "j2Rts" is not a variable we defined
>
> 4.Future plan
> - Display the application or metadata list of a referenced variable in the
> variable basic management module
> - Improve user experience
>
> Best,
> Chunjin Mu
>
> Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:28写道：
>
> > When we have dozens or dozens of variables, this phenomenon may be more
> > prominent
> >
> >
> > Best,
> > Chunjin Mu
> >
> > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:26写道：
> >
> >> When a person asks which kafka should be used, I tell him in Chinese
> that
> >> he should use "采集整合的kafka", in fact the variable code is collect.kafka,
> but
> >> he is searching for gather keyword, it is difficult to just pass
> >> collect.kafka The intuitive understanding is to "采集整合的kafka", because
> >> the information contained in the variable code is still not that much.
> >>
> >> Huajie Wang <be...@gmail.com> 于2022年10月14日周五 22:17写道：
> >>
> >>> I just thought about it, there is really no need for this name.
> >>>
> >>>
> >>>
> >>> Best,
> >>> Huajie Wang
> >>>
> >>>
> >>>
> >>> Rui Fan <19...@gmail.com> 于2022年10月14日周五 21:54写道：
> >>>
> >>> > Hi guys,
> >>> >
> >>> > I understand chunjin's solution is this:
> >>> >
> >>> > - variable code is placeholder
> >>> > - variable name is an introduction
> >>> > - Description is a detailed introduction
> >>> >
> >>> > I think introduction and detail are redundant.
> >>> >
> >>> > For example: there are two kafka clusters, the Beijing kafka
> >>> > cluster and the Hangzhou kafka cluster.
> >>> >
> >>> > If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why
> >>> > does the introduction still need `北京 kafka 集群` and
> >>> > `杭州 kafka 集群`? placeholder is already clear.
> >>> >
> >>> > If dev may forget the placeholder, then dev may also
> >>> > forget the introduction.
> >>> >
> >>> > Of course, if the placeholder is kafka1 kafka2, an
> >>> > introduction is required. Why do we need an introduction
> >>> > at this point, because the placeholder name is unreasonable.
> >>> >
> >>> > Just like variable naming in Java code, there is no need to
> >>> > introduce an introduction with a clear name. If the meaning of
> >>> > the variable is complicated, we also need to write a detailed
> >>> > description.
> >>> >
> >>> > Finally, I want to say: less is more. If we want to make
> >>> > something complicated, please give everyone a strong reason.
> >>> >
> >>> > Best
> >>> > fanrui
> >>> >
> >>> > On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <be...@gmail.com>
> wrote:
> >>> >
> >>> > > Hi guys,
> >>> > >
> >>> > > I agree with Chunjin, I think variable name is useful, Chunjin also
> >>> > > explained the reason.
> >>> > >
> >>> > >
> >>> > > Best,
> >>> > > Huajie Wang
> >>> > >
> >>> > >
> >>> > >
> >>> > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 20:22写道：
> >>> > >
> >>> > > > Hi, Rui Fan
> >>> > > > I'll start a discussion later.
> >>> > > >
> >>> > > > Best
> >>> > > > Chunjin Mu
> >>> > > >
> >>> > > > Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
> >>> > > >
> >>> > > > > Hi guys,
> >>> > > > >
> >>> > > > > I agree with monster. And I suggest use variable name, variable
> >>> value
> >>> > > and
> >>> > > > > description.
> >>> > > > >
> >>> > > > > By the way, there are some problems still need to confirm.
> >>> What’s the
> >>> > > > > placeholder? And how to use them when edit sql or datastream
> job?
> >>> > It’s
> >>> > > > > better to discuss these asap.
> >>> > > > >
> >>> > > > > Best
> >>> > > > > fanrui
> >>> > > > >
> >>> > > > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <macksonmu0@gmail.com
> >
> >>> > wrote:
> >>> > > > >
> >>> > > > > > hi, huajie, fan rui, monster
> >>> > > > > >
> >>> > > > > > Thank you for your suggestions.
> >>> > > > > >
> >>> > > > > > Now please Huajie to confirm how to do it. If remove the
> >>> variable
> >>> > > > name, I
> >>> > > > > > can also accept it here. I will do small secondary
> development
> >>> on
> >>> > our
> >>> > > > > > production system based on the community to meet internal
> >>> needs.
> >>> > > > > >
> >>> > > > > > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五 15:58写道：
> >>> > > > > >
> >>> > > > > > > "外部采集整合kafka" is the Chinese description of variable, so it
> >>> makes
> >>> > > > sense
> >>> > > > > > to
> >>> > > > > > > put it under `Description`.
> >>> > > > > > >
> >>> > > > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
> >>> > > > > > >
> >>> > > > > > > > The variable management module has three properties,
> >>> including
> >>> > > > > variable
> >>> > > > > > > > code, variable name, and description, For example, the
> >>> variable
> >>> > > > code
> >>> > > > > is
> >>> > > > > > > > "external.kafka.collect.brokers", and the variable name
> is
> >>> > > > > > "外部采集整合kafka".
> >>> > > > > > > > When we created the application, I forgot what the
> variable
> >>> > code
> >>> > > > was,
> >>> > > > > > > but I
> >>> > > > > > > > knew it was called "采集整合", so I can search Enter the
> "采集整合"
> >>> > > > keywords
> >>> > > > > in
> >>> > > > > > > the
> >>> > > > > > > > box, and of course you can also search for kafka
> keywords.
> >>> > > > > > > >
> >>> > > > > > > > At the same time, I want to know more detailed
> information
> >>> > about
> >>> > > > this
> >>> > > > > > > > kafka, such as who the operator is, what is the data
> >>> retention
> >>> > > > > period,
> >>> > > > > > > and
> >>> > > > > > > > how big is the cluster size, but I can't use the
> >>> description to
> >>> > > > > search,
> >>> > > > > > > so
> >>> > > > > > > > I can only search or variable code or variable name, and
> >>> the
> >>> > > > variable
> >>> > > > > > > name
> >>> > > > > > > > is also for a simple display.
> >>> > > > > > > >
> >>> > > > > > > > Although external.kafka.collect.brokers already contains
> a
> >>> lot
> >>> > of
> >>> > > > > > > > information, I think this is just a naming convention.
> >>> When we
> >>> > > > > discuss
> >>> > > > > > > this
> >>> > > > > > > > variable, we will discuss it by using the "采集整合 kafka"
> >>> > > > > > > >
> >>> > > > > > > > We can discuss whether there is a problem with this
> design,
> >>> > what
> >>> > > do
> >>> > > > > you
> >>> > > > > > > > think?
> >>> > > > > > > >
> >>> > > > > > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
> >>> > > > > > > >
> >>> > > > > > > > > Hi huajie:
> >>> > > > > > > > >
> >>> > > > > > > > > > If it is manually entered, it will always be treated
> as
> >>> > > > ordinary
> >>> > > > > > > text.
> >>> > > > > > > > >
> >>> > > > > > > > > Users always copy some code from an old job to a new
> job.
> >>> > > > > > > > > I'm afraid it might not meet the user's expectations.
> >>> > > > > > > > >
> >>> > > > > > > > > Best,
> >>> > > > > > > > > fanrui
> >>> > > > > > > > >
> >>> > > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <
> >>> > benjobx@gmail.com>
> >>> > > > > > wrote:
> >>> > > > > > > > >
> >>> > > > > > > > > > hi 田向阳:
> >>> > > > > > > > > >
> >>> > > > > > > > > > The current discussion is about variable management,
> >>> which
> >>> > > is a
> >>> > > > > > > > > relatively
> >>> > > > > > > > > > independent function, No other modules involved, you
> >>> can
> >>> > > check
> >>> > > > > the
> >>> > > > > > > > thread
> >>> > > > > > > > > > [1]
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > > [1] thread:
> >>> > > > > > > > > >
> >>> > > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > > Best,
> >>> > > > > > > > > > Huajie Wang
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> >>> > > > > > > > > >
> >>> > > > > > > > > > > is this module include a resource management cente
> r
> >>> > where
> >>> > > > jars
> >>> > > > > > can
> >>> > > > > > > > be
> >>> > > > > > > > > > > chosen to provide to create a job
> >>> > > > > > > > > > >
> >>> > > > > > > > > > >
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > ---- 回复的原邮件 ----
> >>> > > > > > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> >>> > > > > > > > > > > | 日期 | 2022年10月08日 12:07 |
> >>> > > > > > > > > > > | 收件人 | dev@streampark.apache.org<
> >>> > > dev@streampark.apache.org>
> >>> > > > |
> >>> > > > > > > > > > > | 抄送至 | |
> >>> > > > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable
> >>> management
> >>> > > > > module |
> >>> > > > > > > > > > > hi 阿穆:
> >>> > > > > > > > > > > Thank you for the clarification, My question is
> >>> answered,
> >>> > > but
> >>> > > > > > now I
> >>> > > > > > > > > have
> >>> > > > > > > > > > > one more concern,
> >>> > > > > > > > > > > I'm currently using {var} for variables, and I'm
> >>> worried
> >>> > > that
> >>> > > > > one
> >>> > > > > > > of
> >>> > > > > > > > > the
> >>> > > > > > > > > > > user's parameters includes "{" and "}", e.g :
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > --password 'mypass{123}abc'
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > So how do we parse parse this? How about ${var}
> >>> > > > > > > > > > >
> >>> > > > > > > > > > >
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > Best,
> >>> > > > > > > > > > > Huajie Wang
> >>> > > > > > > > > > >
> >>> > > > > > > > > > >
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六
> 11:36写道：
> >>> > > > > > > > > > >
> >>> > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > Thanks for your reply.
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > I have integrated your replies. I really don't
> pay
> >>> much
> >>> > > > > > attention
> >>> > > > > > > > to
> >>> > > > > > > > > > SQL,
> >>> > > > > > > > > > > > so I will do the following.
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > 1.Each team has independent variables that do not
> >>> > affect
> >>> > > > each
> >>> > > > > > > other
> >>> > > > > > > > > and
> >>> > > > > > > > > > > > are isolated from each other, but the same
> >>> variable is
> >>> > > > unique
> >>> > > > > > > under
> >>> > > > > > > > > the
> >>> > > > > > > > > > > > same team, so you need to specify the team when
> >>> > creating
> >>> > > > > > > variables
> >>> > > > > > > > > > (users
> >>> > > > > > > > > > > > may belong to multiple teams, and the teams of
> the
> >>> > > current
> >>> > > > > user
> >>> > > > > > > > will
> >>> > > > > > > > > be
> >>> > > > > > > > > > > > displayed here for users to choose)
> >>> > > > > > > > > > > > 2.For Flink jar jobs, there are two ways to pass
> >>> > > variables
> >>> > > > > > > > > > > > - By using placeholders, such as adding --kafka
> >>> > {mykafka}
> >>> > > > to
> >>> > > > > > the
> >>> > > > > > > > args
> >>> > > > > > > > > > of
> >>> > > > > > > > > > > > the application, 'mykafka' is the variable code
> of
> >>> a
> >>> > > > variable
> >>> > > > > > > > > > > > - Search by variable code or variable name.
> After a
> >>> > > > variable
> >>> > > > > is
> >>> > > > > > > > > > selected,
> >>> > > > > > > > > > > > the parameter transfer method is displayed
> >>> > > > > > > > > > > > 3.Flink sql will use variables as placeholders,
> >>> E.g:
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > CREATE TABLE KafkaTableA (
> >>> > > > > > > > > > > > `user_id` BIGINT,
> >>> > > > > > > > > > > > `item_id` BIGINT,
> >>> > > > > > > > > > > > `type` STRING
> >>> > > > > > > > > > > > ) WITH (
> >>> > > > > > > > > > > > 'connector' = 'kafka',
> >>> > > > > > > > > > > > 'topic' = 'user_behaviorA',
> >>> > > > > > > > > > > > 'properties.bootstrap.servers' =
> >>> > > > > > > > > '{kafka_clusterA}',
> >>> > > > > > > > > > > --
> >>> > > > > > > > > > > > 'kafka_ ClusterA' is the variable code
> >>> > > > > > > > > > > > 'properties.group.id' =
> 'testGroupA',
> >>> > > > > > > > > > > > 'scan.startup.mode' =
> >>> 'earliest-offset',
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > CREATE TABLE KafkaTableB (
> >>> > > > > > > > > > > > `user_id` BIGINT,
> >>> > > > > > > > > > > > `item_id` BIGINT
> >>> > > > > > > > > > > > ) WITH (
> >>> > > > > > > > > > > > 'connector' = 'kafka',
> >>> > > > > > > > > > > > 'topic' = 'user_behaviorB',
> >>> > > > > > > > > > > > 'properties.bootstrap.servers' =
> >>> > > > > > > > > '{kafka_clusterB}',
> >>> > > > > > > > > > > --
> >>> > > > > > > > > > > > 'kafka_ ClusterB' is the variable code
> >>> > > > > > > > > > > > 'properties.group.id' =
> 'testGroupB',
> >>> > > > > > > > > > > > 'scan.startup.mode' =
> >>> 'earliest-offset',
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id,
> >>> > '{type}'
> >>> > > > > FROM
> >>> > > > > > > > > > > > KafkaTableB -- 'type' is the variable
> >>> code
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > 4.In the future, we can consider setting some
> >>> system
> >>> > > > built-in
> >>> > > > > > > > > variables
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > Best,
> >>> > > > > > > > > > > > ChunJin Mu
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
> >>> > > > 1996fanrui@gmail.com
> >>> > > > > > >
> >>> > > > > > > > > > wrote:
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > > Hi 阿穆:
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > It's a great proposal.
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > For usage scenarios, I strongly recommend
> >>> support
> >>> > > for
> >>> > > > > > > > replacing
> >>> > > > > > > > > > > > variables
> >>> > > > > > > > > > > > > in sql. E.g:
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > CREATE TABLE KafkaTable (
> >>> > > > > > > > > > > > > `user_id` BIGINT,
> >>> > > > > > > > > > > > > `item_id` BIGINT
> >>> > > > > > > > > > > > > ) WITH (
> >>> > > > > > > > > > > > > 'connector' = 'kafka',
> >>> > > > > > > > > > > > > 'topic' = 'user_behavior',
> >>> > > > > > > > > > > > >
> >>> 'properties.bootstrap.servers' =
> >>> > > > > > > > > > > '{kafka_clusterA}',
> >>> > > > > > > > > > > > > 'properties.group.id' =
> >>> > > > 'testGroup',
> >>> > > > > > > > > > > > > 'scan.startup.mode' =
> >>> > > > > 'earliest-offset',
> >>> > > > > > > > > > > > > 'format' = 'csv'
> >>> > > > > > > > > > > > > )
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > Note: 'properties.bootstrap.servers' =
> >>> > > > > '{kafka_clusterA}',
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > For variable management, I have some
> question:
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > 1. Is the parameter used within the team? If
> >>> all
> >>> > > teams
> >>> > > > > > use a
> >>> > > > > > > > > > common
> >>> > > > > > > > > > > > > parameter, it
> >>> > > > > > > > > > > > > needs to be defined for each team, right?
> >>> > > > > > > > > > > > > 2. Which users can modify variables? Can all
> >>> users
> >>> > > > > modify
> >>> > > > > > > all
> >>> > > > > > > > > > > > variables of
> >>> > > > > > > > > > > > > their team?
> >>> > > > > > > > > > > > > After the variable is modified incorrectly,
> >>> many
> >>> > > jobs
> >>> > > > > may
> >>> > > > > > be
> >>> > > > > > > > > > > > affected. So I
> >>> > > > > > > > > > > > > suggest that
> >>> > > > > > > > > > > > > only team admin or system admin have
> >>> permission to
> >>> > > > > modify,
> >>> > > > > > > add
> >>> > > > > > > > > and
> >>> > > > > > > > > > > > delete
> >>> > > > > > > > > > > > > variables.
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > Best
> >>> > > > > > > > > > > > > fanrui
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <
> >>> > > > > benjobx@gmail.com
> >>> > > > > > > >
> >>> > > > > > > > > > wrote:
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > > hi 阿穆:
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > Thanks for starting a very useful
> discussion,
> >>> I
> >>> > > have a
> >>> > > > > > some
> >>> > > > > > > > > > > questions:
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > 1. The access boundary problem of variable
> >>> > > management,
> >>> > > > > is
> >>> > > > > > > this
> >>> > > > > > > > > all
> >>> > > > > > > > > > > > > people in the platform can access these
> >>> variables(
> >>> > > no
> >>> > > > > > matter
> >>> > > > > > > > who
> >>> > > > > > > > > > > > > defines) or that only team members can be
> >>> access
> >>> > > these
> >>> > > > > > > > > variables.
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > 2. There are two usage scenarios for
> variables
> >>> > that
> >>> > > I
> >>> > > > > know
> >>> > > > > > > of,
> >>> > > > > > > > > one
> >>> > > > > > > > > > > is
> >>> > > > > > > > > > > > > in the entry parameter of main, and the
> other
> >>> is
> >>> > in
> >>> > > > > > > flinksql.
> >>> > > > > > > > > Are
> >>> > > > > > > > > > > both
> >>> > > > > > > > > > > > > supported, or only supported main entry ?
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > about first question, my opinion:
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > 1. The variable management is associated
> with
> >>> the
> >>> > > > team.
> >>> > > > > > > After
> >>> > > > > > > > > the
> >>> > > > > > > > > > > user
> >>> > > > > > > > > > > > > logs in to the system, he must select a team
> >>> (a
> >>> > user
> >>> > > > can
> >>> > > > > > > > belong
> >>> > > > > > > > > to
> >>> > > > > > > > > > > one
> >>> > > > > > > > > > > > > or more teams), The variable belongs to a
> team
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > 2. All resources between teams are isolated,
> >>> which
> >>> > > > means
> >>> > > > > > > that
> >>> > > > > > > > > > > > > variables are also isolated. In short,
> >>> members of
> >>> > a
> >>> > > > team
> >>> > > > > > can
> >>> > > > > > > > use
> >>> > > > > > > > > > and
> >>> > > > > > > > > > > > > define these variables, which are invisible
> to
> >>> > other
> >>> > > > > > > members,
> >>> > > > > > > > > and
> >>> > > > > > > > > > > > > other members cannot use these variables.
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > Welcome to your views
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > Best,
> >>> > > > > > > > > > > > > Huajie Wang
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > >
> >>> > > > > > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com>
> >>> > 于2022年10月7日周五
> >>> > > > > > > 14:18写道：
> >>> > > > > > > > > > > >
> >>> > > > > > > > > > > > > > Is it possible to do this variable
> >>> support in
> >>> > > sql
> >>> > > > > as
> >>> > > > > > > well
> >>> > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > 阿穆 <305343894@qq.com.invalid>
> >>> > 于2022年10月7日周五
> >>> > > > > > > 13:54写道：
> >>> > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > > Hi guys,
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > > I want to start a discussion about
> >>> the
> >>> > > > > variable
> >>> > > > > > > > > > management
> >>> > > > > > > > > > > > module. Please
> >>> > > > > > > > > > > > > > > see this issue issue-1477.
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > > Background:
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > > In the actual production
> >>> environment,
> >>> > > Flink
> >>> > > > > jobs
> >>> > > > > > > are
> >>> > > > > > > > > > > > generally complex,
> >>> > > > > > > > > > > > > > > and usually require multiple
> >>> external
> >>> > > > > > components.
> >>> > > > > > > > For
> >>> > > > > > > > > > > > example, Flink jobs
> >>> > > > > > > > > > > > > > > consume data from Kafka, then
> >>> connect
> >>> > > > external
> >>> > > > > > > > > > components
> >>> > > > > > > > > > > > such as HBase
> >>> > > > > > > > > > > > > > or
> >>> > > > > > > > > > > > > > > Redis to obtain additional
> business
> >>> > > > > information,
> >>> > > > > > > and
> >>> > > > > > > > > > then
> >>> > > > > > > > > > > > write it to the
> >>> > > > > > > > > > > > > > > downstream external components.
> >>> There
> >>> > are
> >>> > > > the
> >>> > > > > > > > > following
> >>> > > > > > > > > > > > problems:
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > > 1. The connection information of
> >>> > external
> >>> > > > > > > > components,
> >>> > > > > > > > > > such
> >>> > > > > > > > > > > > as IP, port
> >>> > > > > > > > > > > > > > and
> >>> > > > > > > > > > > > > > > user password, needs to be
> >>> configured in
> >>> > > the
> >>> > > > > > > > > application
> >>> > > > > > > > > > > > args and
> >>> > > > > > > > > > > > > > > transferred to the Flink job, so
> >>> that
> >>> > the
> >>> > > > > > > connection
> >>> > > > > > > > > > > > information of
> >>> > > > > > > > > > > > > > > external components is distributed
> >>> in
> >>> > > > multiple
> >>> > > > > > > > > > > > applications. Once the
> >>> > > > > > > > > > > > > > > connection information of external
> >>> > > > components
> >>> > > > > > > > changes,
> >>> > > > > > > > > > > many
> >>> > > > > > > > > > > > application
> >>> > > > > > > > > > > > > > > args parameters need to be
> modified,
> >>> > which
> >>> > > > > will
> >>> > > > > > > lead
> >>> > > > > > > > > to
> >>> > > > > > > > > > > > high operation
> >>> > > > > > > > > > > > > > and
> >>> > > > > > > > > > > > > > > maintenance costs.
> >>> > > > > > > > > > > > > > > 2. There are many people in the
> team
> >>> > > > > developing
> >>> > > > > > > > > > real-time
> >>> > > > > > > > > > > > computing jobs.
> >>> > > > > > > > > > > > > > > There is no uniform specification
> >>> for
> >>> > the
> >>> > > > > > > connection
> >>> > > > > > > > > > > > information of the
> >>> > > > > > > > > > > > > > > external components passed to the
> >>> job,
> >>> > > > > resulting
> >>> > > > > > > in
> >>> > > > > > > > > > > > different parameter
> >>> > > > > > > > > > > > > > > names of the same component. This
> is
> >>> > > > difficult
> >>> > > > > > to
> >>> > > > > > > > > count
> >>> > > > > > > > > > > > which external
> >>> > > > > > > > > > > > > > > components are dependent.
> >>> > > > > > > > > > > > > > > 3. In production practice, there
> are
> >>> > > usually
> >>> > > > > > > > multiple
> >>> > > > > > > > > > sets
> >>> > > > > > > > > > > > of
> >>> > > > > > > > > > > > > > > environments, such as test
> >>> environment
> >>> > and
> >>> > > > > > > > production
> >>> > > > > > > > > > > > environment. It is
> >>> > > > > > > > > > > > > > > not intuitive to judge whether a
> >>> > component
> >>> > > > > > belongs
> >>> > > > > > > > to
> >>> > > > > > > > > a
> >>> > > > > > > > > > > > test environment
> >>> > > > > > > > > > > > > > or
> >>> > > > > > > > > > > > > > > a production environment simply
> >>> through
> >>> > IP
> >>> > > > and
> >>> > > > > > > > ports.
> >>> > > > > > > > > > Even
> >>> > > > > > > > > > > > if it can be
> >>> > > > > > > > > > > > > > > judged, there are some omissions.
> In
> >>> > this
> >>> > > > way,
> >>> > > > > > the
> >>> > > > > > > > > > > > connection information
> >>> > > > > > > > > > > > > > > online to the production
> >>> environment may
> >>> > > be
> >>> > > > > > > external
> >>> > > > > > > > > > > > components of the
> >>> > > > > > > > > > > > > > test
> >>> > > > > > > > > > > > > > > environment, or components of the
> >>> > > production
> >>> > > > > > > > > environment
> >>> > > > > > > > > > > are
> >>> > > > > > > > > > > > > > inadvertently
> >>> > > > > > > > > > > > > > > configured during testing, leading
> >>> to
> >>> > > > > production
> >>> > > > > > > > > > failures.
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > > Proposal:
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > > 1. Create a unified module to
> manage
> >>> > > > > variables.
> >>> > > > > > > > Users
> >>> > > > > > > > > > can
> >>> > > > > > > > > > > > define their
> >>> > > > > > > > > > > > > > own
> >>> > > > > > > > > > > > > > > variables in this module. Variable
> >>> > > > information
> >>> > > > > > > > > includes
> >>> > > > > > > > > > > > variable code
> >>> > > > > > > > > > > > > > > (passed to Flink program as
> >>> parameter
> >>> > > name),
> >>> > > > > > > > variable
> >>> > > > > > > > > > > > value, variable
> >>> > > > > > > > > > > > > name,
> >>> > > > > > > > > > > > > > > variable description, and which
> >>> > > applications
> >>> > > > > > > depend
> >>> > > > > > > > on
> >>> > > > > > > > > > > > > > > 2. In the same Team module, the
> >>> variable
> >>> > > > code
> >>> > > > > or
> >>> > > > > > > > > > variable
> >>> > > > > > > > > > > > name is unique
> >>> > > > > > > > > > > > > > > - By using placeholders, such as
> >>> adding
> >>> > --
> >>> > > > > kafka
> >>> > > > > > > > > > {mykafka}
> >>> > > > > > > > > > > > to the args of
> >>> > > > > > > > > > > > > > > the application, mykafka is the
> >>> variable
> >>> > > > code
> >>> > > > > > of a
> >>> > > > > > > > > > > variable
> >>> > > > > > > > > > > > > > > - Search by variable code or
> >>> variable
> >>> > > name.
> >>> > > > > > After
> >>> > > > > > > a
> >>> > > > > > > > > > > > variable is selected,
> >>> > > > > > > > > > > > > > > the parameter transfer method is
> >>> > displayed
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > > If you have any suggestions,
> >>> welcome to
> >>> > > > > > > participate
> >>> > > > > > > > in
> >>> > > > > > > > > > the
> >>> > > > > > > > > > > > discussion on
> >>> > > > > > > > > > > > > > > the mail list or issue, I look
> >>> forward
> >>> > to
> >>> > > > your
> >>> > > > > > > > > feedback.
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > >
> >>> > > > > > > > > > > > > > > Best wishes
> >>> > > > > > > > > > > > > >
> >>> > > > > > > > > > >
> >>> > > > > > > > > >
> >>> > > > > > > > >
> >>> > > > > > > >
> >>> > > > > > >
> >>> > > > > >
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Chunjin Mu <ma...@gmail.com>.
Hi guys,
Thanks for the discussion and great suggestions.
Here I summarize the functions to be completed in version 1.2.4
1.The table structure of the variable contains the following main contents
- variable code is placeholder, For example the variable code is
"collect.kafka.cluster", it will be referenced as ${collect.kafka.cluster},
it is unique within a team
- variable name is removed, after discussion, it is not necessarily retained
- variable value is the value of a specific variable, such as ip and port
- description is a detailed introduction
- team is the team the variable belongs to
2.Basic functions of variable management
- Add and modify variables: need to check whether the variable code is
unique under the current team
- Deleting a variable: It is necessary to check whether it has been
referenced by the Application under the team to which the variable belongs.
If there is metadata management in the future, it is also necessary to
check whether it has been referenced by the metadata. If it is referenced,
it will prompt that the variable is referenced and cannot be deleted.
3.References in Flink sql and program args
- In the application, the variables are uniformly referenced in the form of
placeholders ${xxx}, such as in Flink sql, program args
- For a better experience, it is necessary to provide the function of
following the search prompt when entering variable placeholders. For
example, when entering "${", the variable list is displayed, and when
"${kafka" is entered, the display contains variables for kafka strings, in
"variable code" or "description".
- When the focus is lost, it is judged whether there is a variable
placeholder, and if so, a verification button is displayed to prompt the
user to perform verification. When the user clicks the verification button,
the replaced content of the placeholder is displayed, and only the user
clicks the verification button to click the submit button
- If the user enters "okir23${j2Rts}pEt", this placeholder "${j2Rts}" will
not be replaced if "j2Rts" is not a variable we defined
4.Future plan
- Display the application or metadata list of a referenced variable in the
variable basic management module
- Improve user experience
Best,
Chunjin Mu
Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:28写道：
> When we have dozens or dozens of variables, this phenomenon may be more
> prominent
>
>
> Best,
> Chunjin Mu
>
> Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:26写道：
>
>> When a person asks which kafka should be used, I tell him in Chinese that
>> he should use "采集整合的kafka", in fact the variable code is collect.kafka, but
>> he is searching for gather keyword, it is difficult to just pass
>> collect.kafka The intuitive understanding is to "采集整合的kafka", because
>> the information contained in the variable code is still not that much.
>>
>> Huajie Wang <be...@gmail.com> 于2022年10月14日周五 22:17写道：
>>
>>> I just thought about it, there is really no need for this name.
>>>
>>>
>>>
>>> Best,
>>> Huajie Wang
>>>
>>>
>>>
>>> Rui Fan <19...@gmail.com> 于2022年10月14日周五 21:54写道：
>>>
>>> > Hi guys,
>>> >
>>> > I understand chunjin's solution is this:
>>> >
>>> > - variable code is placeholder
>>> > - variable name is an introduction
>>> > - Description is a detailed introduction
>>> >
>>> > I think introduction and detail are redundant.
>>> >
>>> > For example: there are two kafka clusters, the Beijing kafka
>>> > cluster and the Hangzhou kafka cluster.
>>> >
>>> > If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why
>>> > does the introduction still need `北京 kafka 集群` and
>>> > `杭州 kafka 集群`? placeholder is already clear.
>>> >
>>> > If dev may forget the placeholder, then dev may also
>>> > forget the introduction.
>>> >
>>> > Of course, if the placeholder is kafka1 kafka2, an
>>> > introduction is required. Why do we need an introduction
>>> > at this point, because the placeholder name is unreasonable.
>>> >
>>> > Just like variable naming in Java code, there is no need to
>>> > introduce an introduction with a clear name. If the meaning of
>>> > the variable is complicated, we also need to write a detailed
>>> > description.
>>> >
>>> > Finally, I want to say: less is more. If we want to make
>>> > something complicated, please give everyone a strong reason.
>>> >
>>> > Best
>>> > fanrui
>>> >
>>> > On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <be...@gmail.com> wrote:
>>> >
>>> > > Hi guys,
>>> > >
>>> > > I agree with Chunjin, I think variable name is useful, Chunjin also
>>> > > explained the reason.
>>> > >
>>> > >
>>> > > Best,
>>> > > Huajie Wang
>>> > >
>>> > >
>>> > >
>>> > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 20:22写道：
>>> > >
>>> > > > Hi, Rui Fan
>>> > > > I'll start a discussion later.
>>> > > >
>>> > > > Best
>>> > > > Chunjin Mu
>>> > > >
>>> > > > Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
>>> > > >
>>> > > > > Hi guys,
>>> > > > >
>>> > > > > I agree with monster. And I suggest use variable name, variable
>>> value
>>> > > and
>>> > > > > description.
>>> > > > >
>>> > > > > By the way, there are some problems still need to confirm.
>>> What’s the
>>> > > > > placeholder? And how to use them when edit sql or datastream job?
>>> > It’s
>>> > > > > better to discuss these asap.
>>> > > > >
>>> > > > > Best
>>> > > > > fanrui
>>> > > > >
>>> > > > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <ma...@gmail.com>
>>> > wrote:
>>> > > > >
>>> > > > > > hi, huajie, fan rui, monster
>>> > > > > >
>>> > > > > > Thank you for your suggestions.
>>> > > > > >
>>> > > > > > Now please Huajie to confirm how to do it. If remove the
>>> variable
>>> > > > name, I
>>> > > > > > can also accept it here. I will do small secondary development
>>> on
>>> > our
>>> > > > > > production system based on the community to meet internal
>>> needs.
>>> > > > > >
>>> > > > > > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五 15:58写道：
>>> > > > > >
>>> > > > > > > "外部采集整合kafka" is the Chinese description of variable, so it
>>> makes
>>> > > > sense
>>> > > > > > to
>>> > > > > > > put it under `Description`.
>>> > > > > > >
>>> > > > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
>>> > > > > > >
>>> > > > > > > > The variable management module has three properties,
>>> including
>>> > > > > variable
>>> > > > > > > > code, variable name, and description, For example, the
>>> variable
>>> > > > code
>>> > > > > is
>>> > > > > > > > "external.kafka.collect.brokers", and the variable name is
>>> > > > > > "外部采集整合kafka".
>>> > > > > > > > When we created the application, I forgot what the variable
>>> > code
>>> > > > was,
>>> > > > > > > but I
>>> > > > > > > > knew it was called "采集整合", so I can search Enter the "采集整合"
>>> > > > keywords
>>> > > > > in
>>> > > > > > > the
>>> > > > > > > > box, and of course you can also search for kafka keywords.
>>> > > > > > > >
>>> > > > > > > > At the same time, I want to know more detailed information
>>> > about
>>> > > > this
>>> > > > > > > > kafka, such as who the operator is, what is the data
>>> retention
>>> > > > > period,
>>> > > > > > > and
>>> > > > > > > > how big is the cluster size, but I can't use the
>>> description to
>>> > > > > search,
>>> > > > > > > so
>>> > > > > > > > I can only search or variable code or variable name, and
>>> the
>>> > > > variable
>>> > > > > > > name
>>> > > > > > > > is also for a simple display.
>>> > > > > > > >
>>> > > > > > > > Although external.kafka.collect.brokers already contains a
>>> lot
>>> > of
>>> > > > > > > > information, I think this is just a naming convention.
>>> When we
>>> > > > > discuss
>>> > > > > > > this
>>> > > > > > > > variable, we will discuss it by using the "采集整合 kafka"
>>> > > > > > > >
>>> > > > > > > > We can discuss whether there is a problem with this design,
>>> > what
>>> > > do
>>> > > > > you
>>> > > > > > > > think?
>>> > > > > > > >
>>> > > > > > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
>>> > > > > > > >
>>> > > > > > > > > Hi huajie:
>>> > > > > > > > >
>>> > > > > > > > > > If it is manually entered, it will always be treated as
>>> > > > ordinary
>>> > > > > > > text.
>>> > > > > > > > >
>>> > > > > > > > > Users always copy some code from an old job to a new job.
>>> > > > > > > > > I'm afraid it might not meet the user's expectations.
>>> > > > > > > > >
>>> > > > > > > > > Best,
>>> > > > > > > > > fanrui
>>> > > > > > > > >
>>> > > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <
>>> > benjobx@gmail.com>
>>> > > > > > wrote:
>>> > > > > > > > >
>>> > > > > > > > > > hi 田向阳:
>>> > > > > > > > > >
>>> > > > > > > > > > The current discussion is about variable management,
>>> which
>>> > > is a
>>> > > > > > > > > relatively
>>> > > > > > > > > > independent function, No other modules involved, you
>>> can
>>> > > check
>>> > > > > the
>>> > > > > > > > thread
>>> > > > > > > > > > [1]
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > [1] thread:
>>> > > > > > > > > >
>>> > > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > Best,
>>> > > > > > > > > > Huajie Wang
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
>>> > > > > > > > > >
>>> > > > > > > > > > > is this module include a resource management cente r
>>> > where
>>> > > > jars
>>> > > > > > can
>>> > > > > > > > be
>>> > > > > > > > > > > chosen to provide to create a job
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > > ---- 回复的原邮件 ----
>>> > > > > > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
>>> > > > > > > > > > > | 日期 | 2022年10月08日 12:07 |
>>> > > > > > > > > > > | 收件人 | dev@streampark.apache.org<
>>> > > dev@streampark.apache.org>
>>> > > > |
>>> > > > > > > > > > > | 抄送至 | |
>>> > > > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable
>>> management
>>> > > > > module |
>>> > > > > > > > > > > hi 阿穆:
>>> > > > > > > > > > > Thank you for the clarification, My question is
>>> answered,
>>> > > but
>>> > > > > > now I
>>> > > > > > > > > have
>>> > > > > > > > > > > one more concern,
>>> > > > > > > > > > > I'm currently using {var} for variables, and I'm
>>> worried
>>> > > that
>>> > > > > one
>>> > > > > > > of
>>> > > > > > > > > the
>>> > > > > > > > > > > user's parameters includes "{" and "}", e.g :
>>> > > > > > > > > > >
>>> > > > > > > > > > > --password 'mypass{123}abc'
>>> > > > > > > > > > >
>>> > > > > > > > > > > So how do we parse parse this? How about ${var}
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > > Best,
>>> > > > > > > > > > > Huajie Wang
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
>>> > > > > > > > > > >
>>> > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Thanks for your reply.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > I have integrated your replies. I really don't pay
>>> much
>>> > > > > > attention
>>> > > > > > > > to
>>> > > > > > > > > > SQL,
>>> > > > > > > > > > > > so I will do the following.
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > 1.Each team has independent variables that do not
>>> > affect
>>> > > > each
>>> > > > > > > other
>>> > > > > > > > > and
>>> > > > > > > > > > > > are isolated from each other, but the same
>>> variable is
>>> > > > unique
>>> > > > > > > under
>>> > > > > > > > > the
>>> > > > > > > > > > > > same team, so you need to specify the team when
>>> > creating
>>> > > > > > > variables
>>> > > > > > > > > > (users
>>> > > > > > > > > > > > may belong to multiple teams, and the teams of the
>>> > > current
>>> > > > > user
>>> > > > > > > > will
>>> > > > > > > > > be
>>> > > > > > > > > > > > displayed here for users to choose)
>>> > > > > > > > > > > > 2.For Flink jar jobs, there are two ways to pass
>>> > > variables
>>> > > > > > > > > > > > - By using placeholders, such as adding --kafka
>>> > {mykafka}
>>> > > > to
>>> > > > > > the
>>> > > > > > > > args
>>> > > > > > > > > > of
>>> > > > > > > > > > > > the application, 'mykafka' is the variable code of
>>> a
>>> > > > variable
>>> > > > > > > > > > > > - Search by variable code or variable name. After a
>>> > > > variable
>>> > > > > is
>>> > > > > > > > > > selected,
>>> > > > > > > > > > > > the parameter transfer method is displayed
>>> > > > > > > > > > > > 3.Flink sql will use variables as placeholders,
>>> E.g:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > CREATE TABLE KafkaTableA (
>>> > > > > > > > > > > > `user_id` BIGINT,
>>> > > > > > > > > > > > `item_id` BIGINT,
>>> > > > > > > > > > > > `type` STRING
>>> > > > > > > > > > > > ) WITH (
>>> > > > > > > > > > > > 'connector' = 'kafka',
>>> > > > > > > > > > > > 'topic' = 'user_behaviorA',
>>> > > > > > > > > > > > 'properties.bootstrap.servers' =
>>> > > > > > > > > '{kafka_clusterA}',
>>> > > > > > > > > > > --
>>> > > > > > > > > > > > 'kafka_ ClusterA' is the variable code
>>> > > > > > > > > > > > 'properties.group.id' = 'testGroupA',
>>> > > > > > > > > > > > 'scan.startup.mode' =
>>> 'earliest-offset',
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > CREATE TABLE KafkaTableB (
>>> > > > > > > > > > > > `user_id` BIGINT,
>>> > > > > > > > > > > > `item_id` BIGINT
>>> > > > > > > > > > > > ) WITH (
>>> > > > > > > > > > > > 'connector' = 'kafka',
>>> > > > > > > > > > > > 'topic' = 'user_behaviorB',
>>> > > > > > > > > > > > 'properties.bootstrap.servers' =
>>> > > > > > > > > '{kafka_clusterB}',
>>> > > > > > > > > > > --
>>> > > > > > > > > > > > 'kafka_ ClusterB' is the variable code
>>> > > > > > > > > > > > 'properties.group.id' = 'testGroupB',
>>> > > > > > > > > > > > 'scan.startup.mode' =
>>> 'earliest-offset',
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id,
>>> > '{type}'
>>> > > > > FROM
>>> > > > > > > > > > > > KafkaTableB -- 'type' is the variable
>>> code
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > 4.In the future, we can consider setting some
>>> system
>>> > > > built-in
>>> > > > > > > > > variables
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > Best,
>>> > > > > > > > > > > > ChunJin Mu
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
>>> > > > 1996fanrui@gmail.com
>>> > > > > > >
>>> > > > > > > > > > wrote:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > > Hi 阿穆:
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > It's a great proposal.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > For usage scenarios, I strongly recommend
>>> support
>>> > > for
>>> > > > > > > > replacing
>>> > > > > > > > > > > > variables
>>> > > > > > > > > > > > > in sql. E.g:
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > CREATE TABLE KafkaTable (
>>> > > > > > > > > > > > > `user_id` BIGINT,
>>> > > > > > > > > > > > > `item_id` BIGINT
>>> > > > > > > > > > > > > ) WITH (
>>> > > > > > > > > > > > > 'connector' = 'kafka',
>>> > > > > > > > > > > > > 'topic' = 'user_behavior',
>>> > > > > > > > > > > > >
>>> 'properties.bootstrap.servers' =
>>> > > > > > > > > > > '{kafka_clusterA}',
>>> > > > > > > > > > > > > 'properties.group.id' =
>>> > > > 'testGroup',
>>> > > > > > > > > > > > > 'scan.startup.mode' =
>>> > > > > 'earliest-offset',
>>> > > > > > > > > > > > > 'format' = 'csv'
>>> > > > > > > > > > > > > )
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > Note: 'properties.bootstrap.servers' =
>>> > > > > '{kafka_clusterA}',
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > For variable management, I have some question:
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > 1. Is the parameter used within the team? If
>>> all
>>> > > teams
>>> > > > > > use a
>>> > > > > > > > > > common
>>> > > > > > > > > > > > > parameter, it
>>> > > > > > > > > > > > > needs to be defined for each team, right?
>>> > > > > > > > > > > > > 2. Which users can modify variables? Can all
>>> users
>>> > > > > modify
>>> > > > > > > all
>>> > > > > > > > > > > > variables of
>>> > > > > > > > > > > > > their team?
>>> > > > > > > > > > > > > After the variable is modified incorrectly,
>>> many
>>> > > jobs
>>> > > > > may
>>> > > > > > be
>>> > > > > > > > > > > > affected. So I
>>> > > > > > > > > > > > > suggest that
>>> > > > > > > > > > > > > only team admin or system admin have
>>> permission to
>>> > > > > modify,
>>> > > > > > > add
>>> > > > > > > > > and
>>> > > > > > > > > > > > delete
>>> > > > > > > > > > > > > variables.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > Best
>>> > > > > > > > > > > > > fanrui
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <
>>> > > > > benjobx@gmail.com
>>> > > > > > > >
>>> > > > > > > > > > wrote:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > > hi 阿穆:
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > Thanks for starting a very useful discussion,
>>> I
>>> > > have a
>>> > > > > > some
>>> > > > > > > > > > > questions:
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > 1. The access boundary problem of variable
>>> > > management,
>>> > > > > is
>>> > > > > > > this
>>> > > > > > > > > all
>>> > > > > > > > > > > > > people in the platform can access these
>>> variables(
>>> > > no
>>> > > > > > matter
>>> > > > > > > > who
>>> > > > > > > > > > > > > defines) or that only team members can be
>>> access
>>> > > these
>>> > > > > > > > > variables.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > 2. There are two usage scenarios for variables
>>> > that
>>> > > I
>>> > > > > know
>>> > > > > > > of,
>>> > > > > > > > > one
>>> > > > > > > > > > > is
>>> > > > > > > > > > > > > in the entry parameter of main, and the other
>>> is
>>> > in
>>> > > > > > > flinksql.
>>> > > > > > > > > Are
>>> > > > > > > > > > > both
>>> > > > > > > > > > > > > supported, or only supported main entry ?
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > about first question, my opinion:
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > 1. The variable management is associated with
>>> the
>>> > > > team.
>>> > > > > > > After
>>> > > > > > > > > the
>>> > > > > > > > > > > user
>>> > > > > > > > > > > > > logs in to the system, he must select a team
>>> (a
>>> > user
>>> > > > can
>>> > > > > > > > belong
>>> > > > > > > > > to
>>> > > > > > > > > > > one
>>> > > > > > > > > > > > > or more teams), The variable belongs to a team
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > 2. All resources between teams are isolated,
>>> which
>>> > > > means
>>> > > > > > > that
>>> > > > > > > > > > > > > variables are also isolated. In short,
>>> members of
>>> > a
>>> > > > team
>>> > > > > > can
>>> > > > > > > > use
>>> > > > > > > > > > and
>>> > > > > > > > > > > > > define these variables, which are invisible to
>>> > other
>>> > > > > > > members,
>>> > > > > > > > > and
>>> > > > > > > > > > > > > other members cannot use these variables.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > Welcome to your views
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > Best,
>>> > > > > > > > > > > > > Huajie Wang
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com>
>>> > 于2022年10月7日周五
>>> > > > > > > 14:18写道：
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > > > Is it possible to do this variable
>>> support in
>>> > > sql
>>> > > > > as
>>> > > > > > > well
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > 阿穆 <305343894@qq.com.invalid>
>>> > 于2022年10月7日周五
>>> > > > > > > 13:54写道：
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > Hi guys,
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > I want to start a discussion about
>>> the
>>> > > > > variable
>>> > > > > > > > > > management
>>> > > > > > > > > > > > module. Please
>>> > > > > > > > > > > > > > > see this issue issue-1477.
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > Background:
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > In the actual production
>>> environment,
>>> > > Flink
>>> > > > > jobs
>>> > > > > > > are
>>> > > > > > > > > > > > generally complex,
>>> > > > > > > > > > > > > > > and usually require multiple
>>> external
>>> > > > > > components.
>>> > > > > > > > For
>>> > > > > > > > > > > > example, Flink jobs
>>> > > > > > > > > > > > > > > consume data from Kafka, then
>>> connect
>>> > > > external
>>> > > > > > > > > > components
>>> > > > > > > > > > > > such as HBase
>>> > > > > > > > > > > > > > or
>>> > > > > > > > > > > > > > > Redis to obtain additional business
>>> > > > > information,
>>> > > > > > > and
>>> > > > > > > > > > then
>>> > > > > > > > > > > > write it to the
>>> > > > > > > > > > > > > > > downstream external components.
>>> There
>>> > are
>>> > > > the
>>> > > > > > > > > following
>>> > > > > > > > > > > > problems:
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > 1. The connection information of
>>> > external
>>> > > > > > > > components,
>>> > > > > > > > > > such
>>> > > > > > > > > > > > as IP, port
>>> > > > > > > > > > > > > > and
>>> > > > > > > > > > > > > > > user password, needs to be
>>> configured in
>>> > > the
>>> > > > > > > > > application
>>> > > > > > > > > > > > args and
>>> > > > > > > > > > > > > > > transferred to the Flink job, so
>>> that
>>> > the
>>> > > > > > > connection
>>> > > > > > > > > > > > information of
>>> > > > > > > > > > > > > > > external components is distributed
>>> in
>>> > > > multiple
>>> > > > > > > > > > > > applications. Once the
>>> > > > > > > > > > > > > > > connection information of external
>>> > > > components
>>> > > > > > > > changes,
>>> > > > > > > > > > > many
>>> > > > > > > > > > > > application
>>> > > > > > > > > > > > > > > args parameters need to be modified,
>>> > which
>>> > > > > will
>>> > > > > > > lead
>>> > > > > > > > > to
>>> > > > > > > > > > > > high operation
>>> > > > > > > > > > > > > > and
>>> > > > > > > > > > > > > > > maintenance costs.
>>> > > > > > > > > > > > > > > 2. There are many people in the team
>>> > > > > developing
>>> > > > > > > > > > real-time
>>> > > > > > > > > > > > computing jobs.
>>> > > > > > > > > > > > > > > There is no uniform specification
>>> for
>>> > the
>>> > > > > > > connection
>>> > > > > > > > > > > > information of the
>>> > > > > > > > > > > > > > > external components passed to the
>>> job,
>>> > > > > resulting
>>> > > > > > > in
>>> > > > > > > > > > > > different parameter
>>> > > > > > > > > > > > > > > names of the same component. This is
>>> > > > difficult
>>> > > > > > to
>>> > > > > > > > > count
>>> > > > > > > > > > > > which external
>>> > > > > > > > > > > > > > > components are dependent.
>>> > > > > > > > > > > > > > > 3. In production practice, there are
>>> > > usually
>>> > > > > > > > multiple
>>> > > > > > > > > > sets
>>> > > > > > > > > > > > of
>>> > > > > > > > > > > > > > > environments, such as test
>>> environment
>>> > and
>>> > > > > > > > production
>>> > > > > > > > > > > > environment. It is
>>> > > > > > > > > > > > > > > not intuitive to judge whether a
>>> > component
>>> > > > > > belongs
>>> > > > > > > > to
>>> > > > > > > > > a
>>> > > > > > > > > > > > test environment
>>> > > > > > > > > > > > > > or
>>> > > > > > > > > > > > > > > a production environment simply
>>> through
>>> > IP
>>> > > > and
>>> > > > > > > > ports.
>>> > > > > > > > > > Even
>>> > > > > > > > > > > > if it can be
>>> > > > > > > > > > > > > > > judged, there are some omissions. In
>>> > this
>>> > > > way,
>>> > > > > > the
>>> > > > > > > > > > > > connection information
>>> > > > > > > > > > > > > > > online to the production
>>> environment may
>>> > > be
>>> > > > > > > external
>>> > > > > > > > > > > > components of the
>>> > > > > > > > > > > > > > test
>>> > > > > > > > > > > > > > > environment, or components of the
>>> > > production
>>> > > > > > > > > environment
>>> > > > > > > > > > > are
>>> > > > > > > > > > > > > > inadvertently
>>> > > > > > > > > > > > > > > configured during testing, leading
>>> to
>>> > > > > production
>>> > > > > > > > > > failures.
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > Proposal:
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > 1. Create a unified module to manage
>>> > > > > variables.
>>> > > > > > > > Users
>>> > > > > > > > > > can
>>> > > > > > > > > > > > define their
>>> > > > > > > > > > > > > > own
>>> > > > > > > > > > > > > > > variables in this module. Variable
>>> > > > information
>>> > > > > > > > > includes
>>> > > > > > > > > > > > variable code
>>> > > > > > > > > > > > > > > (passed to Flink program as
>>> parameter
>>> > > name),
>>> > > > > > > > variable
>>> > > > > > > > > > > > value, variable
>>> > > > > > > > > > > > > name,
>>> > > > > > > > > > > > > > > variable description, and which
>>> > > applications
>>> > > > > > > depend
>>> > > > > > > > on
>>> > > > > > > > > > > > > > > 2. In the same Team module, the
>>> variable
>>> > > > code
>>> > > > > or
>>> > > > > > > > > > variable
>>> > > > > > > > > > > > name is unique
>>> > > > > > > > > > > > > > > - By using placeholders, such as
>>> adding
>>> > --
>>> > > > > kafka
>>> > > > > > > > > > {mykafka}
>>> > > > > > > > > > > > to the args of
>>> > > > > > > > > > > > > > > the application, mykafka is the
>>> variable
>>> > > > code
>>> > > > > > of a
>>> > > > > > > > > > > variable
>>> > > > > > > > > > > > > > > - Search by variable code or
>>> variable
>>> > > name.
>>> > > > > > After
>>> > > > > > > a
>>> > > > > > > > > > > > variable is selected,
>>> > > > > > > > > > > > > > > the parameter transfer method is
>>> > displayed
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > If you have any suggestions,
>>> welcome to
>>> > > > > > > participate
>>> > > > > > > > in
>>> > > > > > > > > > the
>>> > > > > > > > > > > > discussion on
>>> > > > > > > > > > > > > > > the mail list or issue, I look
>>> forward
>>> > to
>>> > > > your
>>> > > > > > > > > feedback.
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > Best wishes
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Chunjin Mu <ma...@gmail.com>.
When we have dozens or dozens of variables, this phenomenon may be more
prominent
Best,
Chunjin Mu
Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 22:26写道：
> When a person asks which kafka should be used, I tell him in Chinese that
> he should use "采集整合的kafka", in fact the variable code is collect.kafka, but
> he is searching for gather keyword, it is difficult to just pass
> collect.kafka The intuitive understanding is to "采集整合的kafka", because
> the information contained in the variable code is still not that much.
>
> Huajie Wang <be...@gmail.com> 于2022年10月14日周五 22:17写道：
>
>> I just thought about it, there is really no need for this name.
>>
>>
>>
>> Best,
>> Huajie Wang
>>
>>
>>
>> Rui Fan <19...@gmail.com> 于2022年10月14日周五 21:54写道：
>>
>> > Hi guys,
>> >
>> > I understand chunjin's solution is this:
>> >
>> > - variable code is placeholder
>> > - variable name is an introduction
>> > - Description is a detailed introduction
>> >
>> > I think introduction and detail are redundant.
>> >
>> > For example: there are two kafka clusters, the Beijing kafka
>> > cluster and the Hangzhou kafka cluster.
>> >
>> > If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why
>> > does the introduction still need `北京 kafka 集群` and
>> > `杭州 kafka 集群`? placeholder is already clear.
>> >
>> > If dev may forget the placeholder, then dev may also
>> > forget the introduction.
>> >
>> > Of course, if the placeholder is kafka1 kafka2, an
>> > introduction is required. Why do we need an introduction
>> > at this point, because the placeholder name is unreasonable.
>> >
>> > Just like variable naming in Java code, there is no need to
>> > introduce an introduction with a clear name. If the meaning of
>> > the variable is complicated, we also need to write a detailed
>> > description.
>> >
>> > Finally, I want to say: less is more. If we want to make
>> > something complicated, please give everyone a strong reason.
>> >
>> > Best
>> > fanrui
>> >
>> > On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <be...@gmail.com> wrote:
>> >
>> > > Hi guys,
>> > >
>> > > I agree with Chunjin, I think variable name is useful, Chunjin also
>> > > explained the reason.
>> > >
>> > >
>> > > Best,
>> > > Huajie Wang
>> > >
>> > >
>> > >
>> > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 20:22写道：
>> > >
>> > > > Hi, Rui Fan
>> > > > I'll start a discussion later.
>> > > >
>> > > > Best
>> > > > Chunjin Mu
>> > > >
>> > > > Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
>> > > >
>> > > > > Hi guys,
>> > > > >
>> > > > > I agree with monster. And I suggest use variable name, variable
>> value
>> > > and
>> > > > > description.
>> > > > >
>> > > > > By the way, there are some problems still need to confirm. What’s
>> the
>> > > > > placeholder? And how to use them when edit sql or datastream job?
>> > It’s
>> > > > > better to discuss these asap.
>> > > > >
>> > > > > Best
>> > > > > fanrui
>> > > > >
>> > > > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <ma...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > > hi, huajie, fan rui, monster
>> > > > > >
>> > > > > > Thank you for your suggestions.
>> > > > > >
>> > > > > > Now please Huajie to confirm how to do it. If remove the
>> variable
>> > > > name, I
>> > > > > > can also accept it here. I will do small secondary development
>> on
>> > our
>> > > > > > production system based on the community to meet internal needs.
>> > > > > >
>> > > > > > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五 15:58写道：
>> > > > > >
>> > > > > > > "外部采集整合kafka" is the Chinese description of variable, so it
>> makes
>> > > > sense
>> > > > > > to
>> > > > > > > put it under `Description`.
>> > > > > > >
>> > > > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
>> > > > > > >
>> > > > > > > > The variable management module has three properties,
>> including
>> > > > > variable
>> > > > > > > > code, variable name, and description, For example, the
>> variable
>> > > > code
>> > > > > is
>> > > > > > > > "external.kafka.collect.brokers", and the variable name is
>> > > > > > "外部采集整合kafka".
>> > > > > > > > When we created the application, I forgot what the variable
>> > code
>> > > > was,
>> > > > > > > but I
>> > > > > > > > knew it was called "采集整合", so I can search Enter the "采集整合"
>> > > > keywords
>> > > > > in
>> > > > > > > the
>> > > > > > > > box, and of course you can also search for kafka keywords.
>> > > > > > > >
>> > > > > > > > At the same time, I want to know more detailed information
>> > about
>> > > > this
>> > > > > > > > kafka, such as who the operator is, what is the data
>> retention
>> > > > > period,
>> > > > > > > and
>> > > > > > > > how big is the cluster size, but I can't use the
>> description to
>> > > > > search,
>> > > > > > > so
>> > > > > > > > I can only search or variable code or variable name, and the
>> > > > variable
>> > > > > > > name
>> > > > > > > > is also for a simple display.
>> > > > > > > >
>> > > > > > > > Although external.kafka.collect.brokers already contains a
>> lot
>> > of
>> > > > > > > > information, I think this is just a naming convention. When
>> we
>> > > > > discuss
>> > > > > > > this
>> > > > > > > > variable, we will discuss it by using the "采集整合 kafka"
>> > > > > > > >
>> > > > > > > > We can discuss whether there is a problem with this design,
>> > what
>> > > do
>> > > > > you
>> > > > > > > > think?
>> > > > > > > >
>> > > > > > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
>> > > > > > > >
>> > > > > > > > > Hi huajie:
>> > > > > > > > >
>> > > > > > > > > > If it is manually entered, it will always be treated as
>> > > > ordinary
>> > > > > > > text.
>> > > > > > > > >
>> > > > > > > > > Users always copy some code from an old job to a new job.
>> > > > > > > > > I'm afraid it might not meet the user's expectations.
>> > > > > > > > >
>> > > > > > > > > Best,
>> > > > > > > > > fanrui
>> > > > > > > > >
>> > > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <
>> > benjobx@gmail.com>
>> > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > hi 田向阳:
>> > > > > > > > > >
>> > > > > > > > > > The current discussion is about variable management,
>> which
>> > > is a
>> > > > > > > > > relatively
>> > > > > > > > > > independent function, No other modules involved, you can
>> > > check
>> > > > > the
>> > > > > > > > thread
>> > > > > > > > > > [1]
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > [1] thread:
>> > > > > > > > > >
>> > > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Best,
>> > > > > > > > > > Huajie Wang
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
>> > > > > > > > > >
>> > > > > > > > > > > is this module include a resource management cente r
>> > where
>> > > > jars
>> > > > > > can
>> > > > > > > > be
>> > > > > > > > > > > chosen to provide to create a job
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > ---- 回复的原邮件 ----
>> > > > > > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
>> > > > > > > > > > > | 日期 | 2022年10月08日 12:07 |
>> > > > > > > > > > > | 收件人 | dev@streampark.apache.org<
>> > > dev@streampark.apache.org>
>> > > > |
>> > > > > > > > > > > | 抄送至 | |
>> > > > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable
>> management
>> > > > > module |
>> > > > > > > > > > > hi 阿穆:
>> > > > > > > > > > > Thank you for the clarification, My question is
>> answered,
>> > > but
>> > > > > > now I
>> > > > > > > > > have
>> > > > > > > > > > > one more concern,
>> > > > > > > > > > > I'm currently using {var} for variables, and I'm
>> worried
>> > > that
>> > > > > one
>> > > > > > > of
>> > > > > > > > > the
>> > > > > > > > > > > user's parameters includes "{" and "}", e.g :
>> > > > > > > > > > >
>> > > > > > > > > > > --password 'mypass{123}abc'
>> > > > > > > > > > >
>> > > > > > > > > > > So how do we parse parse this? How about ${var}
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > Best,
>> > > > > > > > > > > Huajie Wang
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks for your reply.
>> > > > > > > > > > > >
>> > > > > > > > > > > > I have integrated your replies. I really don't pay
>> much
>> > > > > > attention
>> > > > > > > > to
>> > > > > > > > > > SQL,
>> > > > > > > > > > > > so I will do the following.
>> > > > > > > > > > > >
>> > > > > > > > > > > > 1.Each team has independent variables that do not
>> > affect
>> > > > each
>> > > > > > > other
>> > > > > > > > > and
>> > > > > > > > > > > > are isolated from each other, but the same variable
>> is
>> > > > unique
>> > > > > > > under
>> > > > > > > > > the
>> > > > > > > > > > > > same team, so you need to specify the team when
>> > creating
>> > > > > > > variables
>> > > > > > > > > > (users
>> > > > > > > > > > > > may belong to multiple teams, and the teams of the
>> > > current
>> > > > > user
>> > > > > > > > will
>> > > > > > > > > be
>> > > > > > > > > > > > displayed here for users to choose)
>> > > > > > > > > > > > 2.For Flink jar jobs, there are two ways to pass
>> > > variables
>> > > > > > > > > > > > - By using placeholders, such as adding --kafka
>> > {mykafka}
>> > > > to
>> > > > > > the
>> > > > > > > > args
>> > > > > > > > > > of
>> > > > > > > > > > > > the application, 'mykafka' is the variable code of a
>> > > > variable
>> > > > > > > > > > > > - Search by variable code or variable name. After a
>> > > > variable
>> > > > > is
>> > > > > > > > > > selected,
>> > > > > > > > > > > > the parameter transfer method is displayed
>> > > > > > > > > > > > 3.Flink sql will use variables as placeholders, E.g:
>> > > > > > > > > > > >
>> > > > > > > > > > > > CREATE TABLE KafkaTableA (
>> > > > > > > > > > > > `user_id` BIGINT,
>> > > > > > > > > > > > `item_id` BIGINT,
>> > > > > > > > > > > > `type` STRING
>> > > > > > > > > > > > ) WITH (
>> > > > > > > > > > > > 'connector' = 'kafka',
>> > > > > > > > > > > > 'topic' = 'user_behaviorA',
>> > > > > > > > > > > > 'properties.bootstrap.servers' =
>> > > > > > > > > '{kafka_clusterA}',
>> > > > > > > > > > > --
>> > > > > > > > > > > > 'kafka_ ClusterA' is the variable code
>> > > > > > > > > > > > 'properties.group.id' = 'testGroupA',
>> > > > > > > > > > > > 'scan.startup.mode' =
>> 'earliest-offset',
>> > > > > > > > > > > >
>> > > > > > > > > > > > CREATE TABLE KafkaTableB (
>> > > > > > > > > > > > `user_id` BIGINT,
>> > > > > > > > > > > > `item_id` BIGINT
>> > > > > > > > > > > > ) WITH (
>> > > > > > > > > > > > 'connector' = 'kafka',
>> > > > > > > > > > > > 'topic' = 'user_behaviorB',
>> > > > > > > > > > > > 'properties.bootstrap.servers' =
>> > > > > > > > > '{kafka_clusterB}',
>> > > > > > > > > > > --
>> > > > > > > > > > > > 'kafka_ ClusterB' is the variable code
>> > > > > > > > > > > > 'properties.group.id' = 'testGroupB',
>> > > > > > > > > > > > 'scan.startup.mode' =
>> 'earliest-offset',
>> > > > > > > > > > > >
>> > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id,
>> > '{type}'
>> > > > > FROM
>> > > > > > > > > > > > KafkaTableB -- 'type' is the variable
>> code
>> > > > > > > > > > > >
>> > > > > > > > > > > > 4.In the future, we can consider setting some system
>> > > > built-in
>> > > > > > > > > variables
>> > > > > > > > > > > >
>> > > > > > > > > > > > Best,
>> > > > > > > > > > > > ChunJin Mu
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
>> > > > 1996fanrui@gmail.com
>> > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi 阿穆:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > It's a great proposal.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > For usage scenarios, I strongly recommend
>> support
>> > > for
>> > > > > > > > replacing
>> > > > > > > > > > > > variables
>> > > > > > > > > > > > > in sql. E.g:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > CREATE TABLE KafkaTable (
>> > > > > > > > > > > > > `user_id` BIGINT,
>> > > > > > > > > > > > > `item_id` BIGINT
>> > > > > > > > > > > > > ) WITH (
>> > > > > > > > > > > > > 'connector' = 'kafka',
>> > > > > > > > > > > > > 'topic' = 'user_behavior',
>> > > > > > > > > > > > >
>> 'properties.bootstrap.servers' =
>> > > > > > > > > > > '{kafka_clusterA}',
>> > > > > > > > > > > > > 'properties.group.id' =
>> > > > 'testGroup',
>> > > > > > > > > > > > > 'scan.startup.mode' =
>> > > > > 'earliest-offset',
>> > > > > > > > > > > > > 'format' = 'csv'
>> > > > > > > > > > > > > )
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Note: 'properties.bootstrap.servers' =
>> > > > > '{kafka_clusterA}',
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > For variable management, I have some question:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 1. Is the parameter used within the team? If
>> all
>> > > teams
>> > > > > > use a
>> > > > > > > > > > common
>> > > > > > > > > > > > > parameter, it
>> > > > > > > > > > > > > needs to be defined for each team, right?
>> > > > > > > > > > > > > 2. Which users can modify variables? Can all
>> users
>> > > > > modify
>> > > > > > > all
>> > > > > > > > > > > > variables of
>> > > > > > > > > > > > > their team?
>> > > > > > > > > > > > > After the variable is modified incorrectly,
>> many
>> > > jobs
>> > > > > may
>> > > > > > be
>> > > > > > > > > > > > affected. So I
>> > > > > > > > > > > > > suggest that
>> > > > > > > > > > > > > only team admin or system admin have
>> permission to
>> > > > > modify,
>> > > > > > > add
>> > > > > > > > > and
>> > > > > > > > > > > > delete
>> > > > > > > > > > > > > variables.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best
>> > > > > > > > > > > > > fanrui
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <
>> > > > > benjobx@gmail.com
>> > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > hi 阿穆:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks for starting a very useful discussion, I
>> > > have a
>> > > > > > some
>> > > > > > > > > > > questions:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 1. The access boundary problem of variable
>> > > management,
>> > > > > is
>> > > > > > > this
>> > > > > > > > > all
>> > > > > > > > > > > > > people in the platform can access these
>> variables(
>> > > no
>> > > > > > matter
>> > > > > > > > who
>> > > > > > > > > > > > > defines) or that only team members can be
>> access
>> > > these
>> > > > > > > > > variables.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 2. There are two usage scenarios for variables
>> > that
>> > > I
>> > > > > know
>> > > > > > > of,
>> > > > > > > > > one
>> > > > > > > > > > > is
>> > > > > > > > > > > > > in the entry parameter of main, and the other
>> is
>> > in
>> > > > > > > flinksql.
>> > > > > > > > > Are
>> > > > > > > > > > > both
>> > > > > > > > > > > > > supported, or only supported main entry ?
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > about first question, my opinion:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 1. The variable management is associated with
>> the
>> > > > team.
>> > > > > > > After
>> > > > > > > > > the
>> > > > > > > > > > > user
>> > > > > > > > > > > > > logs in to the system, he must select a team (a
>> > user
>> > > > can
>> > > > > > > > belong
>> > > > > > > > > to
>> > > > > > > > > > > one
>> > > > > > > > > > > > > or more teams), The variable belongs to a team
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 2. All resources between teams are isolated,
>> which
>> > > > means
>> > > > > > > that
>> > > > > > > > > > > > > variables are also isolated. In short, members
>> of
>> > a
>> > > > team
>> > > > > > can
>> > > > > > > > use
>> > > > > > > > > > and
>> > > > > > > > > > > > > define these variables, which are invisible to
>> > other
>> > > > > > > members,
>> > > > > > > > > and
>> > > > > > > > > > > > > other members cannot use these variables.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Welcome to your views
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Best,
>> > > > > > > > > > > > > Huajie Wang
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com>
>> > 于2022年10月7日周五
>> > > > > > > 14:18写道：
>> > > > > > > > > > > >
>> > > > > > > > > > > > > > Is it possible to do this variable
>> support in
>> > > sql
>> > > > > as
>> > > > > > > well
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > 阿穆 <305343894@qq.com.invalid>
>> > 于2022年10月7日周五
>> > > > > > > 13:54写道：
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Hi guys,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > I want to start a discussion about
>> the
>> > > > > variable
>> > > > > > > > > > management
>> > > > > > > > > > > > module. Please
>> > > > > > > > > > > > > > > see this issue issue-1477.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Background:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > In the actual production environment,
>> > > Flink
>> > > > > jobs
>> > > > > > > are
>> > > > > > > > > > > > generally complex,
>> > > > > > > > > > > > > > > and usually require multiple external
>> > > > > > components.
>> > > > > > > > For
>> > > > > > > > > > > > example, Flink jobs
>> > > > > > > > > > > > > > > consume data from Kafka, then connect
>> > > > external
>> > > > > > > > > > components
>> > > > > > > > > > > > such as HBase
>> > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > Redis to obtain additional business
>> > > > > information,
>> > > > > > > and
>> > > > > > > > > > then
>> > > > > > > > > > > > write it to the
>> > > > > > > > > > > > > > > downstream external components. There
>> > are
>> > > > the
>> > > > > > > > > following
>> > > > > > > > > > > > problems:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > 1. The connection information of
>> > external
>> > > > > > > > components,
>> > > > > > > > > > such
>> > > > > > > > > > > > as IP, port
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > user password, needs to be
>> configured in
>> > > the
>> > > > > > > > > application
>> > > > > > > > > > > > args and
>> > > > > > > > > > > > > > > transferred to the Flink job, so that
>> > the
>> > > > > > > connection
>> > > > > > > > > > > > information of
>> > > > > > > > > > > > > > > external components is distributed in
>> > > > multiple
>> > > > > > > > > > > > applications. Once the
>> > > > > > > > > > > > > > > connection information of external
>> > > > components
>> > > > > > > > changes,
>> > > > > > > > > > > many
>> > > > > > > > > > > > application
>> > > > > > > > > > > > > > > args parameters need to be modified,
>> > which
>> > > > > will
>> > > > > > > lead
>> > > > > > > > > to
>> > > > > > > > > > > > high operation
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > maintenance costs.
>> > > > > > > > > > > > > > > 2. There are many people in the team
>> > > > > developing
>> > > > > > > > > > real-time
>> > > > > > > > > > > > computing jobs.
>> > > > > > > > > > > > > > > There is no uniform specification for
>> > the
>> > > > > > > connection
>> > > > > > > > > > > > information of the
>> > > > > > > > > > > > > > > external components passed to the
>> job,
>> > > > > resulting
>> > > > > > > in
>> > > > > > > > > > > > different parameter
>> > > > > > > > > > > > > > > names of the same component. This is
>> > > > difficult
>> > > > > > to
>> > > > > > > > > count
>> > > > > > > > > > > > which external
>> > > > > > > > > > > > > > > components are dependent.
>> > > > > > > > > > > > > > > 3. In production practice, there are
>> > > usually
>> > > > > > > > multiple
>> > > > > > > > > > sets
>> > > > > > > > > > > > of
>> > > > > > > > > > > > > > > environments, such as test
>> environment
>> > and
>> > > > > > > > production
>> > > > > > > > > > > > environment. It is
>> > > > > > > > > > > > > > > not intuitive to judge whether a
>> > component
>> > > > > > belongs
>> > > > > > > > to
>> > > > > > > > > a
>> > > > > > > > > > > > test environment
>> > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > a production environment simply
>> through
>> > IP
>> > > > and
>> > > > > > > > ports.
>> > > > > > > > > > Even
>> > > > > > > > > > > > if it can be
>> > > > > > > > > > > > > > > judged, there are some omissions. In
>> > this
>> > > > way,
>> > > > > > the
>> > > > > > > > > > > > connection information
>> > > > > > > > > > > > > > > online to the production environment
>> may
>> > > be
>> > > > > > > external
>> > > > > > > > > > > > components of the
>> > > > > > > > > > > > > > test
>> > > > > > > > > > > > > > > environment, or components of the
>> > > production
>> > > > > > > > > environment
>> > > > > > > > > > > are
>> > > > > > > > > > > > > > inadvertently
>> > > > > > > > > > > > > > > configured during testing, leading to
>> > > > > production
>> > > > > > > > > > failures.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Proposal:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > 1. Create a unified module to manage
>> > > > > variables.
>> > > > > > > > Users
>> > > > > > > > > > can
>> > > > > > > > > > > > define their
>> > > > > > > > > > > > > > own
>> > > > > > > > > > > > > > > variables in this module. Variable
>> > > > information
>> > > > > > > > > includes
>> > > > > > > > > > > > variable code
>> > > > > > > > > > > > > > > (passed to Flink program as parameter
>> > > name),
>> > > > > > > > variable
>> > > > > > > > > > > > value, variable
>> > > > > > > > > > > > > name,
>> > > > > > > > > > > > > > > variable description, and which
>> > > applications
>> > > > > > > depend
>> > > > > > > > on
>> > > > > > > > > > > > > > > 2. In the same Team module, the
>> variable
>> > > > code
>> > > > > or
>> > > > > > > > > > variable
>> > > > > > > > > > > > name is unique
>> > > > > > > > > > > > > > > - By using placeholders, such as
>> adding
>> > --
>> > > > > kafka
>> > > > > > > > > > {mykafka}
>> > > > > > > > > > > > to the args of
>> > > > > > > > > > > > > > > the application, mykafka is the
>> variable
>> > > > code
>> > > > > > of a
>> > > > > > > > > > > variable
>> > > > > > > > > > > > > > > - Search by variable code or variable
>> > > name.
>> > > > > > After
>> > > > > > > a
>> > > > > > > > > > > > variable is selected,
>> > > > > > > > > > > > > > > the parameter transfer method is
>> > displayed
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > If you have any suggestions, welcome
>> to
>> > > > > > > participate
>> > > > > > > > in
>> > > > > > > > > > the
>> > > > > > > > > > > > discussion on
>> > > > > > > > > > > > > > > the mail list or issue, I look
>> forward
>> > to
>> > > > your
>> > > > > > > > > feedback.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Best wishes
>> > > > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Chunjin Mu <ma...@gmail.com>.
When a person asks which kafka should be used, I tell him in Chinese that
he should use "采集整合的kafka", in fact the variable code is collect.kafka, but
he is searching for gather keyword, it is difficult to just pass
collect.kafka The intuitive understanding is to "采集整合的kafka", because the
information contained in the variable code is still not that much.
Huajie Wang <be...@gmail.com> 于2022年10月14日周五 22:17写道：
> I just thought about it, there is really no need for this name.
>
>
>
> Best,
> Huajie Wang
>
>
>
> Rui Fan <19...@gmail.com> 于2022年10月14日周五 21:54写道：
>
> > Hi guys,
> >
> > I understand chunjin's solution is this:
> >
> > - variable code is placeholder
> > - variable name is an introduction
> > - Description is a detailed introduction
> >
> > I think introduction and detail are redundant.
> >
> > For example: there are two kafka clusters, the Beijing kafka
> > cluster and the Hangzhou kafka cluster.
> >
> > If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why
> > does the introduction still need `北京 kafka 集群` and
> > `杭州 kafka 集群`? placeholder is already clear.
> >
> > If dev may forget the placeholder, then dev may also
> > forget the introduction.
> >
> > Of course, if the placeholder is kafka1 kafka2, an
> > introduction is required. Why do we need an introduction
> > at this point, because the placeholder name is unreasonable.
> >
> > Just like variable naming in Java code, there is no need to
> > introduce an introduction with a clear name. If the meaning of
> > the variable is complicated, we also need to write a detailed
> > description.
> >
> > Finally, I want to say: less is more. If we want to make
> > something complicated, please give everyone a strong reason.
> >
> > Best
> > fanrui
> >
> > On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <be...@gmail.com> wrote:
> >
> > > Hi guys,
> > >
> > > I agree with Chunjin, I think variable name is useful, Chunjin also
> > > explained the reason.
> > >
> > >
> > > Best,
> > > Huajie Wang
> > >
> > >
> > >
> > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 20:22写道：
> > >
> > > > Hi, Rui Fan
> > > > I'll start a discussion later.
> > > >
> > > > Best
> > > > Chunjin Mu
> > > >
> > > > Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
> > > >
> > > > > Hi guys,
> > > > >
> > > > > I agree with monster. And I suggest use variable name, variable
> value
> > > and
> > > > > description.
> > > > >
> > > > > By the way, there are some problems still need to confirm. What’s
> the
> > > > > placeholder? And how to use them when edit sql or datastream job?
> > It’s
> > > > > better to discuss these asap.
> > > > >
> > > > > Best
> > > > > fanrui
> > > > >
> > > > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <ma...@gmail.com>
> > wrote:
> > > > >
> > > > > > hi, huajie, fan rui, monster
> > > > > >
> > > > > > Thank you for your suggestions.
> > > > > >
> > > > > > Now please Huajie to confirm how to do it. If remove the variable
> > > > name, I
> > > > > > can also accept it here. I will do small secondary development on
> > our
> > > > > > production system based on the community to meet internal needs.
> > > > > >
> > > > > > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五 15:58写道：
> > > > > >
> > > > > > > "外部采集整合kafka" is the Chinese description of variable, so it
> makes
> > > > sense
> > > > > > to
> > > > > > > put it under `Description`.
> > > > > > >
> > > > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
> > > > > > >
> > > > > > > > The variable management module has three properties,
> including
> > > > > variable
> > > > > > > > code, variable name, and description, For example, the
> variable
> > > > code
> > > > > is
> > > > > > > > "external.kafka.collect.brokers", and the variable name is
> > > > > > "外部采集整合kafka".
> > > > > > > > When we created the application, I forgot what the variable
> > code
> > > > was,
> > > > > > > but I
> > > > > > > > knew it was called "采集整合", so I can search Enter the "采集整合"
> > > > keywords
> > > > > in
> > > > > > > the
> > > > > > > > box, and of course you can also search for kafka keywords.
> > > > > > > >
> > > > > > > > At the same time, I want to know more detailed information
> > about
> > > > this
> > > > > > > > kafka, such as who the operator is, what is the data
> retention
> > > > > period,
> > > > > > > and
> > > > > > > > how big is the cluster size, but I can't use the description
> to
> > > > > search,
> > > > > > > so
> > > > > > > > I can only search or variable code or variable name, and the
> > > > variable
> > > > > > > name
> > > > > > > > is also for a simple display.
> > > > > > > >
> > > > > > > > Although external.kafka.collect.brokers already contains a
> lot
> > of
> > > > > > > > information, I think this is just a naming convention. When
> we
> > > > > discuss
> > > > > > > this
> > > > > > > > variable, we will discuss it by using the "采集整合 kafka"
> > > > > > > >
> > > > > > > > We can discuss whether there is a problem with this design,
> > what
> > > do
> > > > > you
> > > > > > > > think?
> > > > > > > >
> > > > > > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
> > > > > > > >
> > > > > > > > > Hi huajie:
> > > > > > > > >
> > > > > > > > > > If it is manually entered, it will always be treated as
> > > > ordinary
> > > > > > > text.
> > > > > > > > >
> > > > > > > > > Users always copy some code from an old job to a new job.
> > > > > > > > > I'm afraid it might not meet the user's expectations.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > fanrui
> > > > > > > > >
> > > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <
> > benjobx@gmail.com>
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > hi 田向阳:
> > > > > > > > > >
> > > > > > > > > > The current discussion is about variable management,
> which
> > > is a
> > > > > > > > > relatively
> > > > > > > > > > independent function, No other modules involved, you can
> > > check
> > > > > the
> > > > > > > > thread
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > [1] thread:
> > > > > > > > > >
> > > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Huajie Wang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > > > > > > > > >
> > > > > > > > > > > is this module include a resource management cente r
> > where
> > > > jars
> > > > > > can
> > > > > > > > be
> > > > > > > > > > > chosen to provide to create a job
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > ---- 回复的原邮件 ----
> > > > > > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > > > > > > > > > | 日期 | 2022年10月08日 12:07 |
> > > > > > > > > > > | 收件人 | dev@streampark.apache.org<
> > > dev@streampark.apache.org>
> > > > |
> > > > > > > > > > > | 抄送至 | |
> > > > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable
> management
> > > > > module |
> > > > > > > > > > > hi 阿穆:
> > > > > > > > > > > Thank you for the clarification, My question is
> answered,
> > > but
> > > > > > now I
> > > > > > > > > have
> > > > > > > > > > > one more concern,
> > > > > > > > > > > I'm currently using {var} for variables, and I'm
> worried
> > > that
> > > > > one
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > user's parameters includes "{" and "}", e.g :
> > > > > > > > > > >
> > > > > > > > > > > --password 'mypass{123}abc'
> > > > > > > > > > >
> > > > > > > > > > > So how do we parse parse this? How about ${var}
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Huajie Wang
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> > > > > > > > > > >
> > > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > > >
> > > > > > > > > > > > I have integrated your replies. I really don't pay
> much
> > > > > > attention
> > > > > > > > to
> > > > > > > > > > SQL,
> > > > > > > > > > > > so I will do the following.
> > > > > > > > > > > >
> > > > > > > > > > > > 1.Each team has independent variables that do not
> > affect
> > > > each
> > > > > > > other
> > > > > > > > > and
> > > > > > > > > > > > are isolated from each other, but the same variable
> is
> > > > unique
> > > > > > > under
> > > > > > > > > the
> > > > > > > > > > > > same team, so you need to specify the team when
> > creating
> > > > > > > variables
> > > > > > > > > > (users
> > > > > > > > > > > > may belong to multiple teams, and the teams of the
> > > current
> > > > > user
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > > > displayed here for users to choose)
> > > > > > > > > > > > 2.For Flink jar jobs, there are two ways to pass
> > > variables
> > > > > > > > > > > > - By using placeholders, such as adding --kafka
> > {mykafka}
> > > > to
> > > > > > the
> > > > > > > > args
> > > > > > > > > > of
> > > > > > > > > > > > the application, 'mykafka' is the variable code of a
> > > > variable
> > > > > > > > > > > > - Search by variable code or variable name. After a
> > > > variable
> > > > > is
> > > > > > > > > > selected,
> > > > > > > > > > > > the parameter transfer method is displayed
> > > > > > > > > > > > 3.Flink sql will use variables as placeholders, E.g:
> > > > > > > > > > > >
> > > > > > > > > > > > CREATE TABLE KafkaTableA (
> > > > > > > > > > > > `user_id` BIGINT,
> > > > > > > > > > > > `item_id` BIGINT,
> > > > > > > > > > > > `type` STRING
> > > > > > > > > > > > ) WITH (
> > > > > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > > > > 'topic' = 'user_behaviorA',
> > > > > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > > > > '{kafka_clusterA}',
> > > > > > > > > > > --
> > > > > > > > > > > > 'kafka_ ClusterA' is the variable code
> > > > > > > > > > > > 'properties.group.id' = 'testGroupA',
> > > > > > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > > > > > >
> > > > > > > > > > > > CREATE TABLE KafkaTableB (
> > > > > > > > > > > > `user_id` BIGINT,
> > > > > > > > > > > > `item_id` BIGINT
> > > > > > > > > > > > ) WITH (
> > > > > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > > > > 'topic' = 'user_behaviorB',
> > > > > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > > > > '{kafka_clusterB}',
> > > > > > > > > > > --
> > > > > > > > > > > > 'kafka_ ClusterB' is the variable code
> > > > > > > > > > > > 'properties.group.id' = 'testGroupB',
> > > > > > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > > > > > >
> > > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id,
> > '{type}'
> > > > > FROM
> > > > > > > > > > > > KafkaTableB -- 'type' is the variable
> code
> > > > > > > > > > > >
> > > > > > > > > > > > 4.In the future, we can consider setting some system
> > > > built-in
> > > > > > > > > variables
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > ChunJin Mu
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
> > > > 1996fanrui@gmail.com
> > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi 阿穆:
> > > > > > > > > > > > >
> > > > > > > > > > > > > It's a great proposal.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For usage scenarios, I strongly recommend
> support
> > > for
> > > > > > > > replacing
> > > > > > > > > > > > variables
> > > > > > > > > > > > > in sql. E.g:
> > > > > > > > > > > > >
> > > > > > > > > > > > > CREATE TABLE KafkaTable (
> > > > > > > > > > > > > `user_id` BIGINT,
> > > > > > > > > > > > > `item_id` BIGINT
> > > > > > > > > > > > > ) WITH (
> > > > > > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > > > > > 'topic' = 'user_behavior',
> > > > > > > > > > > > >
> 'properties.bootstrap.servers' =
> > > > > > > > > > > '{kafka_clusterA}',
> > > > > > > > > > > > > 'properties.group.id' =
> > > > 'testGroup',
> > > > > > > > > > > > > 'scan.startup.mode' =
> > > > > 'earliest-offset',
> > > > > > > > > > > > > 'format' = 'csv'
> > > > > > > > > > > > > )
> > > > > > > > > > > > >
> > > > > > > > > > > > > Note: 'properties.bootstrap.servers' =
> > > > > '{kafka_clusterA}',
> > > > > > > > > > > > >
> > > > > > > > > > > > > For variable management, I have some question:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Is the parameter used within the team? If all
> > > teams
> > > > > > use a
> > > > > > > > > > common
> > > > > > > > > > > > > parameter, it
> > > > > > > > > > > > > needs to be defined for each team, right?
> > > > > > > > > > > > > 2. Which users can modify variables? Can all
> users
> > > > > modify
> > > > > > > all
> > > > > > > > > > > > variables of
> > > > > > > > > > > > > their team?
> > > > > > > > > > > > > After the variable is modified incorrectly, many
> > > jobs
> > > > > may
> > > > > > be
> > > > > > > > > > > > affected. So I
> > > > > > > > > > > > > suggest that
> > > > > > > > > > > > > only team admin or system admin have permission
> to
> > > > > modify,
> > > > > > > add
> > > > > > > > > and
> > > > > > > > > > > > delete
> > > > > > > > > > > > > variables.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best
> > > > > > > > > > > > > fanrui
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <
> > > > > benjobx@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > hi 阿穆:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for starting a very useful discussion, I
> > > have a
> > > > > > some
> > > > > > > > > > > questions:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. The access boundary problem of variable
> > > management,
> > > > > is
> > > > > > > this
> > > > > > > > > all
> > > > > > > > > > > > > people in the platform can access these
> variables(
> > > no
> > > > > > matter
> > > > > > > > who
> > > > > > > > > > > > > defines) or that only team members can be access
> > > these
> > > > > > > > > variables.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. There are two usage scenarios for variables
> > that
> > > I
> > > > > know
> > > > > > > of,
> > > > > > > > > one
> > > > > > > > > > > is
> > > > > > > > > > > > > in the entry parameter of main, and the other is
> > in
> > > > > > > flinksql.
> > > > > > > > > Are
> > > > > > > > > > > both
> > > > > > > > > > > > > supported, or only supported main entry ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > about first question, my opinion:
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. The variable management is associated with
> the
> > > > team.
> > > > > > > After
> > > > > > > > > the
> > > > > > > > > > > user
> > > > > > > > > > > > > logs in to the system, he must select a team (a
> > user
> > > > can
> > > > > > > > belong
> > > > > > > > > to
> > > > > > > > > > > one
> > > > > > > > > > > > > or more teams), The variable belongs to a team
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2. All resources between teams are isolated,
> which
> > > > means
> > > > > > > that
> > > > > > > > > > > > > variables are also isolated. In short, members
> of
> > a
> > > > team
> > > > > > can
> > > > > > > > use
> > > > > > > > > > and
> > > > > > > > > > > > > define these variables, which are invisible to
> > other
> > > > > > > members,
> > > > > > > > > and
> > > > > > > > > > > > > other members cannot use these variables.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Welcome to your views
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Huajie Wang
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com>
> > 于2022年10月7日周五
> > > > > > > 14:18写道：
> > > > > > > > > > > >
> > > > > > > > > > > > > > Is it possible to do this variable support
> in
> > > sql
> > > > > as
> > > > > > > well
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 阿穆 <305343894@qq.com.invalid>
> > 于2022年10月7日周五
> > > > > > > 13:54写道：
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi guys,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I want to start a discussion about the
> > > > > variable
> > > > > > > > > > management
> > > > > > > > > > > > module. Please
> > > > > > > > > > > > > > > see this issue issue-1477.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Background:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In the actual production environment,
> > > Flink
> > > > > jobs
> > > > > > > are
> > > > > > > > > > > > generally complex,
> > > > > > > > > > > > > > > and usually require multiple external
> > > > > > components.
> > > > > > > > For
> > > > > > > > > > > > example, Flink jobs
> > > > > > > > > > > > > > > consume data from Kafka, then connect
> > > > external
> > > > > > > > > > components
> > > > > > > > > > > > such as HBase
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > Redis to obtain additional business
> > > > > information,
> > > > > > > and
> > > > > > > > > > then
> > > > > > > > > > > > write it to the
> > > > > > > > > > > > > > > downstream external components. There
> > are
> > > > the
> > > > > > > > > following
> > > > > > > > > > > > problems:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. The connection information of
> > external
> > > > > > > > components,
> > > > > > > > > > such
> > > > > > > > > > > > as IP, port
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > user password, needs to be configured
> in
> > > the
> > > > > > > > > application
> > > > > > > > > > > > args and
> > > > > > > > > > > > > > > transferred to the Flink job, so that
> > the
> > > > > > > connection
> > > > > > > > > > > > information of
> > > > > > > > > > > > > > > external components is distributed in
> > > > multiple
> > > > > > > > > > > > applications. Once the
> > > > > > > > > > > > > > > connection information of external
> > > > components
> > > > > > > > changes,
> > > > > > > > > > > many
> > > > > > > > > > > > application
> > > > > > > > > > > > > > > args parameters need to be modified,
> > which
> > > > > will
> > > > > > > lead
> > > > > > > > > to
> > > > > > > > > > > > high operation
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > maintenance costs.
> > > > > > > > > > > > > > > 2. There are many people in the team
> > > > > developing
> > > > > > > > > > real-time
> > > > > > > > > > > > computing jobs.
> > > > > > > > > > > > > > > There is no uniform specification for
> > the
> > > > > > > connection
> > > > > > > > > > > > information of the
> > > > > > > > > > > > > > > external components passed to the job,
> > > > > resulting
> > > > > > > in
> > > > > > > > > > > > different parameter
> > > > > > > > > > > > > > > names of the same component. This is
> > > > difficult
> > > > > > to
> > > > > > > > > count
> > > > > > > > > > > > which external
> > > > > > > > > > > > > > > components are dependent.
> > > > > > > > > > > > > > > 3. In production practice, there are
> > > usually
> > > > > > > > multiple
> > > > > > > > > > sets
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > environments, such as test environment
> > and
> > > > > > > > production
> > > > > > > > > > > > environment. It is
> > > > > > > > > > > > > > > not intuitive to judge whether a
> > component
> > > > > > belongs
> > > > > > > > to
> > > > > > > > > a
> > > > > > > > > > > > test environment
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > a production environment simply
> through
> > IP
> > > > and
> > > > > > > > ports.
> > > > > > > > > > Even
> > > > > > > > > > > > if it can be
> > > > > > > > > > > > > > > judged, there are some omissions. In
> > this
> > > > way,
> > > > > > the
> > > > > > > > > > > > connection information
> > > > > > > > > > > > > > > online to the production environment
> may
> > > be
> > > > > > > external
> > > > > > > > > > > > components of the
> > > > > > > > > > > > > > test
> > > > > > > > > > > > > > > environment, or components of the
> > > production
> > > > > > > > > environment
> > > > > > > > > > > are
> > > > > > > > > > > > > > inadvertently
> > > > > > > > > > > > > > > configured during testing, leading to
> > > > > production
> > > > > > > > > > failures.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Proposal:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 1. Create a unified module to manage
> > > > > variables.
> > > > > > > > Users
> > > > > > > > > > can
> > > > > > > > > > > > define their
> > > > > > > > > > > > > > own
> > > > > > > > > > > > > > > variables in this module. Variable
> > > > information
> > > > > > > > > includes
> > > > > > > > > > > > variable code
> > > > > > > > > > > > > > > (passed to Flink program as parameter
> > > name),
> > > > > > > > variable
> > > > > > > > > > > > value, variable
> > > > > > > > > > > > > name,
> > > > > > > > > > > > > > > variable description, and which
> > > applications
> > > > > > > depend
> > > > > > > > on
> > > > > > > > > > > > > > > 2. In the same Team module, the
> variable
> > > > code
> > > > > or
> > > > > > > > > > variable
> > > > > > > > > > > > name is unique
> > > > > > > > > > > > > > > - By using placeholders, such as
> adding
> > --
> > > > > kafka
> > > > > > > > > > {mykafka}
> > > > > > > > > > > > to the args of
> > > > > > > > > > > > > > > the application, mykafka is the
> variable
> > > > code
> > > > > > of a
> > > > > > > > > > > variable
> > > > > > > > > > > > > > > - Search by variable code or variable
> > > name.
> > > > > > After
> > > > > > > a
> > > > > > > > > > > > variable is selected,
> > > > > > > > > > > > > > > the parameter transfer method is
> > displayed
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If you have any suggestions, welcome
> to
> > > > > > > participate
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > discussion on
> > > > > > > > > > > > > > > the mail list or issue, I look forward
> > to
> > > > your
> > > > > > > > > feedback.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best wishes
> > > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Huajie Wang <be...@gmail.com>.
I just thought about it, there is really no need for this name.
Best,
Huajie Wang
Rui Fan <19...@gmail.com> 于2022年10月14日周五 21:54写道：
> Hi guys,
>
> I understand chunjin's solution is this:
>
> - variable code is placeholder
> - variable name is an introduction
> - Description is a detailed introduction
>
> I think introduction and detail are redundant.
>
> For example: there are two kafka clusters, the Beijing kafka
> cluster and the Hangzhou kafka cluster.
>
> If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why
> does the introduction still need `北京 kafka 集群` and
> `杭州 kafka 集群`? placeholder is already clear.
>
> If dev may forget the placeholder, then dev may also
> forget the introduction.
>
> Of course, if the placeholder is kafka1 kafka2, an
> introduction is required. Why do we need an introduction
> at this point, because the placeholder name is unreasonable.
>
> Just like variable naming in Java code, there is no need to
> introduce an introduction with a clear name. If the meaning of
> the variable is complicated, we also need to write a detailed
> description.
>
> Finally, I want to say: less is more. If we want to make
> something complicated, please give everyone a strong reason.
>
> Best
> fanrui
>
> On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <be...@gmail.com> wrote:
>
> > Hi guys,
> >
> > I agree with Chunjin, I think variable name is useful, Chunjin also
> > explained the reason.
> >
> >
> > Best,
> > Huajie Wang
> >
> >
> >
> > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 20:22写道：
> >
> > > Hi, Rui Fan
> > > I'll start a discussion later.
> > >
> > > Best
> > > Chunjin Mu
> > >
> > > Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
> > >
> > > > Hi guys,
> > > >
> > > > I agree with monster. And I suggest use variable name, variable value
> > and
> > > > description.
> > > >
> > > > By the way, there are some problems still need to confirm. What’s the
> > > > placeholder? And how to use them when edit sql or datastream job?
> It’s
> > > > better to discuss these asap.
> > > >
> > > > Best
> > > > fanrui
> > > >
> > > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <ma...@gmail.com>
> wrote:
> > > >
> > > > > hi, huajie, fan rui, monster
> > > > >
> > > > > Thank you for your suggestions.
> > > > >
> > > > > Now please Huajie to confirm how to do it. If remove the variable
> > > name, I
> > > > > can also accept it here. I will do small secondary development on
> our
> > > > > production system based on the community to meet internal needs.
> > > > >
> > > > > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五 15:58写道：
> > > > >
> > > > > > "外部采集整合kafka" is the Chinese description of variable, so it makes
> > > sense
> > > > > to
> > > > > > put it under `Description`.
> > > > > >
> > > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
> > > > > >
> > > > > > > The variable management module has three properties, including
> > > > variable
> > > > > > > code, variable name, and description, For example, the variable
> > > code
> > > > is
> > > > > > > "external.kafka.collect.brokers", and the variable name is
> > > > > "外部采集整合kafka".
> > > > > > > When we created the application, I forgot what the variable
> code
> > > was,
> > > > > > but I
> > > > > > > knew it was called "采集整合", so I can search Enter the "采集整合"
> > > keywords
> > > > in
> > > > > > the
> > > > > > > box, and of course you can also search for kafka keywords.
> > > > > > >
> > > > > > > At the same time, I want to know more detailed information
> about
> > > this
> > > > > > > kafka, such as who the operator is, what is the data retention
> > > > period,
> > > > > > and
> > > > > > > how big is the cluster size, but I can't use the description to
> > > > search,
> > > > > > so
> > > > > > > I can only search or variable code or variable name, and the
> > > variable
> > > > > > name
> > > > > > > is also for a simple display.
> > > > > > >
> > > > > > > Although external.kafka.collect.brokers already contains a lot
> of
> > > > > > > information, I think this is just a naming convention. When we
> > > > discuss
> > > > > > this
> > > > > > > variable, we will discuss it by using the "采集整合 kafka"
> > > > > > >
> > > > > > > We can discuss whether there is a problem with this design,
> what
> > do
> > > > you
> > > > > > > think?
> > > > > > >
> > > > > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
> > > > > > >
> > > > > > > > Hi huajie:
> > > > > > > >
> > > > > > > > > If it is manually entered, it will always be treated as
> > > ordinary
> > > > > > text.
> > > > > > > >
> > > > > > > > Users always copy some code from an old job to a new job.
> > > > > > > > I'm afraid it might not meet the user's expectations.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > fanrui
> > > > > > > >
> > > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <
> benjobx@gmail.com>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > hi 田向阳:
> > > > > > > > >
> > > > > > > > > The current discussion is about variable management, which
> > is a
> > > > > > > > relatively
> > > > > > > > > independent function, No other modules involved, you can
> > check
> > > > the
> > > > > > > thread
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [1] thread:
> > > > > > > > >
> > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Huajie Wang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > > > > > > > >
> > > > > > > > > > is this module include a resource management cente r
> where
> > > jars
> > > > > can
> > > > > > > be
> > > > > > > > > > chosen to provide to create a job
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > ---- 回复的原邮件 ----
> > > > > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > > > > > > > > | 日期 | 2022年10月08日 12:07 |
> > > > > > > > > > | 收件人 | dev@streampark.apache.org<
> > dev@streampark.apache.org>
> > > |
> > > > > > > > > > | 抄送至 | |
> > > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management
> > > > module |
> > > > > > > > > > hi 阿穆:
> > > > > > > > > > Thank you for the clarification, My question is answered,
> > but
> > > > > now I
> > > > > > > > have
> > > > > > > > > > one more concern,
> > > > > > > > > > I'm currently using {var} for variables, and I'm worried
> > that
> > > > one
> > > > > > of
> > > > > > > > the
> > > > > > > > > > user's parameters includes "{" and "}", e.g :
> > > > > > > > > >
> > > > > > > > > > --password 'mypass{123}abc'
> > > > > > > > > >
> > > > > > > > > > So how do we parse parse this? How about ${var}
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Huajie Wang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> > > > > > > > > >
> > > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your reply.
> > > > > > > > > > >
> > > > > > > > > > > I have integrated your replies. I really don't pay much
> > > > > attention
> > > > > > > to
> > > > > > > > > SQL,
> > > > > > > > > > > so I will do the following.
> > > > > > > > > > >
> > > > > > > > > > > 1.Each team has independent variables that do not
> affect
> > > each
> > > > > > other
> > > > > > > > and
> > > > > > > > > > > are isolated from each other, but the same variable is
> > > unique
> > > > > > under
> > > > > > > > the
> > > > > > > > > > > same team, so you need to specify the team when
> creating
> > > > > > variables
> > > > > > > > > (users
> > > > > > > > > > > may belong to multiple teams, and the teams of the
> > current
> > > > user
> > > > > > > will
> > > > > > > > be
> > > > > > > > > > > displayed here for users to choose)
> > > > > > > > > > > 2.For Flink jar jobs, there are two ways to pass
> > variables
> > > > > > > > > > > - By using placeholders, such as adding --kafka
> {mykafka}
> > > to
> > > > > the
> > > > > > > args
> > > > > > > > > of
> > > > > > > > > > > the application, 'mykafka' is the variable code of a
> > > variable
> > > > > > > > > > > - Search by variable code or variable name. After a
> > > variable
> > > > is
> > > > > > > > > selected,
> > > > > > > > > > > the parameter transfer method is displayed
> > > > > > > > > > > 3.Flink sql will use variables as placeholders, E.g:
> > > > > > > > > > >
> > > > > > > > > > > CREATE TABLE KafkaTableA (
> > > > > > > > > > > `user_id` BIGINT,
> > > > > > > > > > > `item_id` BIGINT,
> > > > > > > > > > > `type` STRING
> > > > > > > > > > > ) WITH (
> > > > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > > > 'topic' = 'user_behaviorA',
> > > > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > > > '{kafka_clusterA}',
> > > > > > > > > > --
> > > > > > > > > > > 'kafka_ ClusterA' is the variable code
> > > > > > > > > > > 'properties.group.id' = 'testGroupA',
> > > > > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > > > > >
> > > > > > > > > > > CREATE TABLE KafkaTableB (
> > > > > > > > > > > `user_id` BIGINT,
> > > > > > > > > > > `item_id` BIGINT
> > > > > > > > > > > ) WITH (
> > > > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > > > 'topic' = 'user_behaviorB',
> > > > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > > > '{kafka_clusterB}',
> > > > > > > > > > --
> > > > > > > > > > > 'kafka_ ClusterB' is the variable code
> > > > > > > > > > > 'properties.group.id' = 'testGroupB',
> > > > > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > > > > >
> > > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id,
> '{type}'
> > > > FROM
> > > > > > > > > > > KafkaTableB -- 'type' is the variable code
> > > > > > > > > > >
> > > > > > > > > > > 4.In the future, we can consider setting some system
> > > built-in
> > > > > > > > variables
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > ChunJin Mu
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
> > > 1996fanrui@gmail.com
> > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi 阿穆:
> > > > > > > > > > > >
> > > > > > > > > > > > It's a great proposal.
> > > > > > > > > > > >
> > > > > > > > > > > > For usage scenarios, I strongly recommend support
> > for
> > > > > > > replacing
> > > > > > > > > > > variables
> > > > > > > > > > > > in sql. E.g:
> > > > > > > > > > > >
> > > > > > > > > > > > CREATE TABLE KafkaTable (
> > > > > > > > > > > > `user_id` BIGINT,
> > > > > > > > > > > > `item_id` BIGINT
> > > > > > > > > > > > ) WITH (
> > > > > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > > > > 'topic' = 'user_behavior',
> > > > > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > > > > > '{kafka_clusterA}',
> > > > > > > > > > > > 'properties.group.id' =
> > > 'testGroup',
> > > > > > > > > > > > 'scan.startup.mode' =
> > > > 'earliest-offset',
> > > > > > > > > > > > 'format' = 'csv'
> > > > > > > > > > > > )
> > > > > > > > > > > >
> > > > > > > > > > > > Note: 'properties.bootstrap.servers' =
> > > > '{kafka_clusterA}',
> > > > > > > > > > > >
> > > > > > > > > > > > For variable management, I have some question:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Is the parameter used within the team? If all
> > teams
> > > > > use a
> > > > > > > > > common
> > > > > > > > > > > > parameter, it
> > > > > > > > > > > > needs to be defined for each team, right?
> > > > > > > > > > > > 2. Which users can modify variables? Can all users
> > > > modify
> > > > > > all
> > > > > > > > > > > variables of
> > > > > > > > > > > > their team?
> > > > > > > > > > > > After the variable is modified incorrectly, many
> > jobs
> > > > may
> > > > > be
> > > > > > > > > > > affected. So I
> > > > > > > > > > > > suggest that
> > > > > > > > > > > > only team admin or system admin have permission to
> > > > modify,
> > > > > > add
> > > > > > > > and
> > > > > > > > > > > delete
> > > > > > > > > > > > variables.
> > > > > > > > > > > >
> > > > > > > > > > > > Best
> > > > > > > > > > > > fanrui
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <
> > > > benjobx@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > hi 阿穆:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for starting a very useful discussion, I
> > have a
> > > > > some
> > > > > > > > > > questions:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 1. The access boundary problem of variable
> > management,
> > > > is
> > > > > > this
> > > > > > > > all
> > > > > > > > > > > > people in the platform can access these variables(
> > no
> > > > > matter
> > > > > > > who
> > > > > > > > > > > > defines) or that only team members can be access
> > these
> > > > > > > > variables.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. There are two usage scenarios for variables
> that
> > I
> > > > know
> > > > > > of,
> > > > > > > > one
> > > > > > > > > > is
> > > > > > > > > > > > in the entry parameter of main, and the other is
> in
> > > > > > flinksql.
> > > > > > > > Are
> > > > > > > > > > both
> > > > > > > > > > > > supported, or only supported main entry ?
> > > > > > > > > > > >
> > > > > > > > > > > > about first question, my opinion:
> > > > > > > > > > > >
> > > > > > > > > > > > 1. The variable management is associated with the
> > > team.
> > > > > > After
> > > > > > > > the
> > > > > > > > > > user
> > > > > > > > > > > > logs in to the system, he must select a team (a
> user
> > > can
> > > > > > > belong
> > > > > > > > to
> > > > > > > > > > one
> > > > > > > > > > > > or more teams), The variable belongs to a team
> > > > > > > > > > > >
> > > > > > > > > > > > 2. All resources between teams are isolated, which
> > > means
> > > > > > that
> > > > > > > > > > > > variables are also isolated. In short, members of
> a
> > > team
> > > > > can
> > > > > > > use
> > > > > > > > > and
> > > > > > > > > > > > define these variables, which are invisible to
> other
> > > > > > members,
> > > > > > > > and
> > > > > > > > > > > > other members cannot use these variables.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Welcome to your views
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Huajie Wang
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com>
> 于2022年10月7日周五
> > > > > > 14:18写道：
> > > > > > > > > > >
> > > > > > > > > > > > > Is it possible to do this variable support in
> > sql
> > > > as
> > > > > > well
> > > > > > > > > > > > >
> > > > > > > > > > > > > 阿穆 <305343894@qq.com.invalid>
> 于2022年10月7日周五
> > > > > > 13:54写道：
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi guys,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I want to start a discussion about the
> > > > variable
> > > > > > > > > management
> > > > > > > > > > > module. Please
> > > > > > > > > > > > > > see this issue issue-1477.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Background:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In the actual production environment,
> > Flink
> > > > jobs
> > > > > > are
> > > > > > > > > > > generally complex,
> > > > > > > > > > > > > > and usually require multiple external
> > > > > components.
> > > > > > > For
> > > > > > > > > > > example, Flink jobs
> > > > > > > > > > > > > > consume data from Kafka, then connect
> > > external
> > > > > > > > > components
> > > > > > > > > > > such as HBase
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > Redis to obtain additional business
> > > > information,
> > > > > > and
> > > > > > > > > then
> > > > > > > > > > > write it to the
> > > > > > > > > > > > > > downstream external components. There
> are
> > > the
> > > > > > > > following
> > > > > > > > > > > problems:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. The connection information of
> external
> > > > > > > components,
> > > > > > > > > such
> > > > > > > > > > > as IP, port
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > user password, needs to be configured in
> > the
> > > > > > > > application
> > > > > > > > > > > args and
> > > > > > > > > > > > > > transferred to the Flink job, so that
> the
> > > > > > connection
> > > > > > > > > > > information of
> > > > > > > > > > > > > > external components is distributed in
> > > multiple
> > > > > > > > > > > applications. Once the
> > > > > > > > > > > > > > connection information of external
> > > components
> > > > > > > changes,
> > > > > > > > > > many
> > > > > > > > > > > application
> > > > > > > > > > > > > > args parameters need to be modified,
> which
> > > > will
> > > > > > lead
> > > > > > > > to
> > > > > > > > > > > high operation
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > maintenance costs.
> > > > > > > > > > > > > > 2. There are many people in the team
> > > > developing
> > > > > > > > > real-time
> > > > > > > > > > > computing jobs.
> > > > > > > > > > > > > > There is no uniform specification for
> the
> > > > > > connection
> > > > > > > > > > > information of the
> > > > > > > > > > > > > > external components passed to the job,
> > > > resulting
> > > > > > in
> > > > > > > > > > > different parameter
> > > > > > > > > > > > > > names of the same component. This is
> > > difficult
> > > > > to
> > > > > > > > count
> > > > > > > > > > > which external
> > > > > > > > > > > > > > components are dependent.
> > > > > > > > > > > > > > 3. In production practice, there are
> > usually
> > > > > > > multiple
> > > > > > > > > sets
> > > > > > > > > > > of
> > > > > > > > > > > > > > environments, such as test environment
> and
> > > > > > > production
> > > > > > > > > > > environment. It is
> > > > > > > > > > > > > > not intuitive to judge whether a
> component
> > > > > belongs
> > > > > > > to
> > > > > > > > a
> > > > > > > > > > > test environment
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > a production environment simply through
> IP
> > > and
> > > > > > > ports.
> > > > > > > > > Even
> > > > > > > > > > > if it can be
> > > > > > > > > > > > > > judged, there are some omissions. In
> this
> > > way,
> > > > > the
> > > > > > > > > > > connection information
> > > > > > > > > > > > > > online to the production environment may
> > be
> > > > > > external
> > > > > > > > > > > components of the
> > > > > > > > > > > > > test
> > > > > > > > > > > > > > environment, or components of the
> > production
> > > > > > > > environment
> > > > > > > > > > are
> > > > > > > > > > > > > inadvertently
> > > > > > > > > > > > > > configured during testing, leading to
> > > > production
> > > > > > > > > failures.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Proposal:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. Create a unified module to manage
> > > > variables.
> > > > > > > Users
> > > > > > > > > can
> > > > > > > > > > > define their
> > > > > > > > > > > > > own
> > > > > > > > > > > > > > variables in this module. Variable
> > > information
> > > > > > > > includes
> > > > > > > > > > > variable code
> > > > > > > > > > > > > > (passed to Flink program as parameter
> > name),
> > > > > > > variable
> > > > > > > > > > > value, variable
> > > > > > > > > > > > name,
> > > > > > > > > > > > > > variable description, and which
> > applications
> > > > > > depend
> > > > > > > on
> > > > > > > > > > > > > > 2. In the same Team module, the variable
> > > code
> > > > or
> > > > > > > > > variable
> > > > > > > > > > > name is unique
> > > > > > > > > > > > > > - By using placeholders, such as adding
> --
> > > > kafka
> > > > > > > > > {mykafka}
> > > > > > > > > > > to the args of
> > > > > > > > > > > > > > the application, mykafka is the variable
> > > code
> > > > > of a
> > > > > > > > > > variable
> > > > > > > > > > > > > > - Search by variable code or variable
> > name.
> > > > > After
> > > > > > a
> > > > > > > > > > > variable is selected,
> > > > > > > > > > > > > > the parameter transfer method is
> displayed
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If you have any suggestions, welcome to
> > > > > > participate
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > discussion on
> > > > > > > > > > > > > > the mail list or issue, I look forward
> to
> > > your
> > > > > > > > feedback.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best wishes
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Rui Fan <19...@gmail.com>.
Hi guys,
I understand chunjin's solution is this:
- variable code is placeholder
- variable name is an introduction
- Description is a detailed introduction
I think introduction and detail are redundant.
For example: there are two kafka clusters, the Beijing kafka
cluster and the Hangzhou kafka cluster.
If placeholder is `beijing_kafka`, `hangzhou_kafka`. Why
does the introduction still need `北京 kafka 集群` and
`杭州 kafka 集群`? placeholder is already clear.
If dev may forget the placeholder, then dev may also
forget the introduction.
Of course, if the placeholder is kafka1 kafka2, an
introduction is required. Why do we need an introduction
at this point, because the placeholder name is unreasonable.
Just like variable naming in Java code, there is no need to
introduce an introduction with a clear name. If the meaning of
the variable is complicated, we also need to write a detailed
description.
Finally, I want to say: less is more. If we want to make
something complicated, please give everyone a strong reason.
Best
fanrui
On Fri, Oct 14, 2022 at 8:42 PM Huajie Wang <be...@gmail.com> wrote:
> Hi guys,
>
> I agree with Chunjin, I think variable name is useful, Chunjin also
> explained the reason.
>
>
> Best,
> Huajie Wang
>
>
>
> Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 20:22写道：
>
> > Hi, Rui Fan
> > I'll start a discussion later.
> >
> > Best
> > Chunjin Mu
> >
> > Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
> >
> > > Hi guys,
> > >
> > > I agree with monster. And I suggest use variable name, variable value
> and
> > > description.
> > >
> > > By the way, there are some problems still need to confirm. What’s the
> > > placeholder? And how to use them when edit sql or datastream job? It’s
> > > better to discuss these asap.
> > >
> > > Best
> > > fanrui
> > >
> > > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <ma...@gmail.com> wrote:
> > >
> > > > hi, huajie, fan rui, monster
> > > >
> > > > Thank you for your suggestions.
> > > >
> > > > Now please Huajie to confirm how to do it. If remove the variable
> > name, I
> > > > can also accept it here. I will do small secondary development on our
> > > > production system based on the community to meet internal needs.
> > > >
> > > > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五 15:58写道：
> > > >
> > > > > "外部采集整合kafka" is the Chinese description of variable, so it makes
> > sense
> > > > to
> > > > > put it under `Description`.
> > > > >
> > > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
> > > > >
> > > > > > The variable management module has three properties, including
> > > variable
> > > > > > code, variable name, and description, For example, the variable
> > code
> > > is
> > > > > > "external.kafka.collect.brokers", and the variable name is
> > > > "外部采集整合kafka".
> > > > > > When we created the application, I forgot what the variable code
> > was,
> > > > > but I
> > > > > > knew it was called "采集整合", so I can search Enter the "采集整合"
> > keywords
> > > in
> > > > > the
> > > > > > box, and of course you can also search for kafka keywords.
> > > > > >
> > > > > > At the same time, I want to know more detailed information about
> > this
> > > > > > kafka, such as who the operator is, what is the data retention
> > > period,
> > > > > and
> > > > > > how big is the cluster size, but I can't use the description to
> > > search,
> > > > > so
> > > > > > I can only search or variable code or variable name, and the
> > variable
> > > > > name
> > > > > > is also for a simple display.
> > > > > >
> > > > > > Although external.kafka.collect.brokers already contains a lot of
> > > > > > information, I think this is just a naming convention. When we
> > > discuss
> > > > > this
> > > > > > variable, we will discuss it by using the "采集整合 kafka"
> > > > > >
> > > > > > We can discuss whether there is a problem with this design, what
> do
> > > you
> > > > > > think?
> > > > > >
> > > > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
> > > > > >
> > > > > > > Hi huajie:
> > > > > > >
> > > > > > > > If it is manually entered, it will always be treated as
> > ordinary
> > > > > text.
> > > > > > >
> > > > > > > Users always copy some code from an old job to a new job.
> > > > > > > I'm afraid it might not meet the user's expectations.
> > > > > > >
> > > > > > > Best,
> > > > > > > fanrui
> > > > > > >
> > > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <be...@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > hi 田向阳:
> > > > > > > >
> > > > > > > > The current discussion is about variable management, which
> is a
> > > > > > > relatively
> > > > > > > > independent function, No other modules involved, you can
> check
> > > the
> > > > > > thread
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > > > [1] thread:
> > > > > > > >
> > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Huajie Wang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > > > > > > >
> > > > > > > > > is this module include a resource management cente r where
> > jars
> > > > can
> > > > > > be
> > > > > > > > > chosen to provide to create a job
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ---- 回复的原邮件 ----
> > > > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > > > > > > > | 日期 | 2022年10月08日 12:07 |
> > > > > > > > > | 收件人 | dev@streampark.apache.org<
> dev@streampark.apache.org>
> > |
> > > > > > > > > | 抄送至 | |
> > > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management
> > > module |
> > > > > > > > > hi 阿穆:
> > > > > > > > > Thank you for the clarification, My question is answered,
> but
> > > > now I
> > > > > > > have
> > > > > > > > > one more concern,
> > > > > > > > > I'm currently using {var} for variables, and I'm worried
> that
> > > one
> > > > > of
> > > > > > > the
> > > > > > > > > user's parameters includes "{" and "}", e.g :
> > > > > > > > >
> > > > > > > > > --password 'mypass{123}abc'
> > > > > > > > >
> > > > > > > > > So how do we parse parse this? How about ${var}
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Huajie Wang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> > > > > > > > >
> > > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > > > > > > > >
> > > > > > > > > > Thanks for your reply.
> > > > > > > > > >
> > > > > > > > > > I have integrated your replies. I really don't pay much
> > > > attention
> > > > > > to
> > > > > > > > SQL,
> > > > > > > > > > so I will do the following.
> > > > > > > > > >
> > > > > > > > > > 1.Each team has independent variables that do not affect
> > each
> > > > > other
> > > > > > > and
> > > > > > > > > > are isolated from each other, but the same variable is
> > unique
> > > > > under
> > > > > > > the
> > > > > > > > > > same team, so you need to specify the team when creating
> > > > > variables
> > > > > > > > (users
> > > > > > > > > > may belong to multiple teams, and the teams of the
> current
> > > user
> > > > > > will
> > > > > > > be
> > > > > > > > > > displayed here for users to choose)
> > > > > > > > > > 2.For Flink jar jobs, there are two ways to pass
> variables
> > > > > > > > > > - By using placeholders, such as adding --kafka {mykafka}
> > to
> > > > the
> > > > > > args
> > > > > > > > of
> > > > > > > > > > the application, 'mykafka' is the variable code of a
> > variable
> > > > > > > > > > - Search by variable code or variable name. After a
> > variable
> > > is
> > > > > > > > selected,
> > > > > > > > > > the parameter transfer method is displayed
> > > > > > > > > > 3.Flink sql will use variables as placeholders, E.g:
> > > > > > > > > >
> > > > > > > > > > CREATE TABLE KafkaTableA (
> > > > > > > > > > `user_id` BIGINT,
> > > > > > > > > > `item_id` BIGINT,
> > > > > > > > > > `type` STRING
> > > > > > > > > > ) WITH (
> > > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > > 'topic' = 'user_behaviorA',
> > > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > > '{kafka_clusterA}',
> > > > > > > > > --
> > > > > > > > > > 'kafka_ ClusterA' is the variable code
> > > > > > > > > > 'properties.group.id' = 'testGroupA',
> > > > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > > > >
> > > > > > > > > > CREATE TABLE KafkaTableB (
> > > > > > > > > > `user_id` BIGINT,
> > > > > > > > > > `item_id` BIGINT
> > > > > > > > > > ) WITH (
> > > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > > 'topic' = 'user_behaviorB',
> > > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > > '{kafka_clusterB}',
> > > > > > > > > --
> > > > > > > > > > 'kafka_ ClusterB' is the variable code
> > > > > > > > > > 'properties.group.id' = 'testGroupB',
> > > > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > > > >
> > > > > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}'
> > > FROM
> > > > > > > > > > KafkaTableB -- 'type' is the variable code
> > > > > > > > > >
> > > > > > > > > > 4.In the future, we can consider setting some system
> > built-in
> > > > > > > variables
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > ChunJin Mu
> > > > > > > > > >
> > > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
> > 1996fanrui@gmail.com
> > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi 阿穆:
> > > > > > > > > > >
> > > > > > > > > > > It's a great proposal.
> > > > > > > > > > >
> > > > > > > > > > > For usage scenarios, I strongly recommend support
> for
> > > > > > replacing
> > > > > > > > > > variables
> > > > > > > > > > > in sql. E.g:
> > > > > > > > > > >
> > > > > > > > > > > CREATE TABLE KafkaTable (
> > > > > > > > > > > `user_id` BIGINT,
> > > > > > > > > > > `item_id` BIGINT
> > > > > > > > > > > ) WITH (
> > > > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > > > 'topic' = 'user_behavior',
> > > > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > > > > '{kafka_clusterA}',
> > > > > > > > > > > 'properties.group.id' =
> > 'testGroup',
> > > > > > > > > > > 'scan.startup.mode' =
> > > 'earliest-offset',
> > > > > > > > > > > 'format' = 'csv'
> > > > > > > > > > > )
> > > > > > > > > > >
> > > > > > > > > > > Note: 'properties.bootstrap.servers' =
> > > '{kafka_clusterA}',
> > > > > > > > > > >
> > > > > > > > > > > For variable management, I have some question:
> > > > > > > > > > >
> > > > > > > > > > > 1. Is the parameter used within the team? If all
> teams
> > > > use a
> > > > > > > > common
> > > > > > > > > > > parameter, it
> > > > > > > > > > > needs to be defined for each team, right?
> > > > > > > > > > > 2. Which users can modify variables? Can all users
> > > modify
> > > > > all
> > > > > > > > > > variables of
> > > > > > > > > > > their team?
> > > > > > > > > > > After the variable is modified incorrectly, many
> jobs
> > > may
> > > > be
> > > > > > > > > > affected. So I
> > > > > > > > > > > suggest that
> > > > > > > > > > > only team admin or system admin have permission to
> > > modify,
> > > > > add
> > > > > > > and
> > > > > > > > > > delete
> > > > > > > > > > > variables.
> > > > > > > > > > >
> > > > > > > > > > > Best
> > > > > > > > > > > fanrui
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <
> > > benjobx@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > hi 阿穆:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks for starting a very useful discussion, I
> have a
> > > > some
> > > > > > > > > questions:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 1. The access boundary problem of variable
> management,
> > > is
> > > > > this
> > > > > > > all
> > > > > > > > > > > people in the platform can access these variables(
> no
> > > > matter
> > > > > > who
> > > > > > > > > > > defines) or that only team members can be access
> these
> > > > > > > variables.
> > > > > > > > > > >
> > > > > > > > > > > 2. There are two usage scenarios for variables that
> I
> > > know
> > > > > of,
> > > > > > > one
> > > > > > > > > is
> > > > > > > > > > > in the entry parameter of main, and the other is in
> > > > > flinksql.
> > > > > > > Are
> > > > > > > > > both
> > > > > > > > > > > supported, or only supported main entry ?
> > > > > > > > > > >
> > > > > > > > > > > about first question, my opinion:
> > > > > > > > > > >
> > > > > > > > > > > 1. The variable management is associated with the
> > team.
> > > > > After
> > > > > > > the
> > > > > > > > > user
> > > > > > > > > > > logs in to the system, he must select a team (a user
> > can
> > > > > > belong
> > > > > > > to
> > > > > > > > > one
> > > > > > > > > > > or more teams), The variable belongs to a team
> > > > > > > > > > >
> > > > > > > > > > > 2. All resources between teams are isolated, which
> > means
> > > > > that
> > > > > > > > > > > variables are also isolated. In short, members of a
> > team
> > > > can
> > > > > > use
> > > > > > > > and
> > > > > > > > > > > define these variables, which are invisible to other
> > > > > members,
> > > > > > > and
> > > > > > > > > > > other members cannot use these variables.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Welcome to your views
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Huajie Wang
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com> 于2022年10月7日周五
> > > > > 14:18写道：
> > > > > > > > > >
> > > > > > > > > > > > Is it possible to do this variable support in
> sql
> > > as
> > > > > well
> > > > > > > > > > > >
> > > > > > > > > > > > 阿穆 <305343894@qq.com.invalid> 于2022年10月7日周五
> > > > > 13:54写道：
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi guys,
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > I want to start a discussion about the
> > > variable
> > > > > > > > management
> > > > > > > > > > module. Please
> > > > > > > > > > > > > see this issue issue-1477.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Background:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > In the actual production environment,
> Flink
> > > jobs
> > > > > are
> > > > > > > > > > generally complex,
> > > > > > > > > > > > > and usually require multiple external
> > > > components.
> > > > > > For
> > > > > > > > > > example, Flink jobs
> > > > > > > > > > > > > consume data from Kafka, then connect
> > external
> > > > > > > > components
> > > > > > > > > > such as HBase
> > > > > > > > > > > > or
> > > > > > > > > > > > > Redis to obtain additional business
> > > information,
> > > > > and
> > > > > > > > then
> > > > > > > > > > write it to the
> > > > > > > > > > > > > downstream external components. There are
> > the
> > > > > > > following
> > > > > > > > > > problems:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. The connection information of external
> > > > > > components,
> > > > > > > > such
> > > > > > > > > > as IP, port
> > > > > > > > > > > > and
> > > > > > > > > > > > > user password, needs to be configured in
> the
> > > > > > > application
> > > > > > > > > > args and
> > > > > > > > > > > > > transferred to the Flink job, so that the
> > > > > connection
> > > > > > > > > > information of
> > > > > > > > > > > > > external components is distributed in
> > multiple
> > > > > > > > > > applications. Once the
> > > > > > > > > > > > > connection information of external
> > components
> > > > > > changes,
> > > > > > > > > many
> > > > > > > > > > application
> > > > > > > > > > > > > args parameters need to be modified, which
> > > will
> > > > > lead
> > > > > > > to
> > > > > > > > > > high operation
> > > > > > > > > > > > and
> > > > > > > > > > > > > maintenance costs.
> > > > > > > > > > > > > 2. There are many people in the team
> > > developing
> > > > > > > > real-time
> > > > > > > > > > computing jobs.
> > > > > > > > > > > > > There is no uniform specification for the
> > > > > connection
> > > > > > > > > > information of the
> > > > > > > > > > > > > external components passed to the job,
> > > resulting
> > > > > in
> > > > > > > > > > different parameter
> > > > > > > > > > > > > names of the same component. This is
> > difficult
> > > > to
> > > > > > > count
> > > > > > > > > > which external
> > > > > > > > > > > > > components are dependent.
> > > > > > > > > > > > > 3. In production practice, there are
> usually
> > > > > > multiple
> > > > > > > > sets
> > > > > > > > > > of
> > > > > > > > > > > > > environments, such as test environment and
> > > > > > production
> > > > > > > > > > environment. It is
> > > > > > > > > > > > > not intuitive to judge whether a component
> > > > belongs
> > > > > > to
> > > > > > > a
> > > > > > > > > > test environment
> > > > > > > > > > > > or
> > > > > > > > > > > > > a production environment simply through IP
> > and
> > > > > > ports.
> > > > > > > > Even
> > > > > > > > > > if it can be
> > > > > > > > > > > > > judged, there are some omissions. In this
> > way,
> > > > the
> > > > > > > > > > connection information
> > > > > > > > > > > > > online to the production environment may
> be
> > > > > external
> > > > > > > > > > components of the
> > > > > > > > > > > > test
> > > > > > > > > > > > > environment, or components of the
> production
> > > > > > > environment
> > > > > > > > > are
> > > > > > > > > > > > inadvertently
> > > > > > > > > > > > > configured during testing, leading to
> > > production
> > > > > > > > failures.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Proposal:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. Create a unified module to manage
> > > variables.
> > > > > > Users
> > > > > > > > can
> > > > > > > > > > define their
> > > > > > > > > > > > own
> > > > > > > > > > > > > variables in this module. Variable
> > information
> > > > > > > includes
> > > > > > > > > > variable code
> > > > > > > > > > > > > (passed to Flink program as parameter
> name),
> > > > > > variable
> > > > > > > > > > value, variable
> > > > > > > > > > > name,
> > > > > > > > > > > > > variable description, and which
> applications
> > > > > depend
> > > > > > on
> > > > > > > > > > > > > 2. In the same Team module, the variable
> > code
> > > or
> > > > > > > > variable
> > > > > > > > > > name is unique
> > > > > > > > > > > > > - By using placeholders, such as adding --
> > > kafka
> > > > > > > > {mykafka}
> > > > > > > > > > to the args of
> > > > > > > > > > > > > the application, mykafka is the variable
> > code
> > > > of a
> > > > > > > > > variable
> > > > > > > > > > > > > - Search by variable code or variable
> name.
> > > > After
> > > > > a
> > > > > > > > > > variable is selected,
> > > > > > > > > > > > > the parameter transfer method is displayed
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > If you have any suggestions, welcome to
> > > > > participate
> > > > > > in
> > > > > > > > the
> > > > > > > > > > discussion on
> > > > > > > > > > > > > the mail list or issue, I look forward to
> > your
> > > > > > > feedback.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best wishes
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Huajie Wang <be...@gmail.com>.
Hi guys,
I agree with Chunjin, I think variable name is useful, Chunjin also
explained the reason.
Best,
Huajie Wang
Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 20:22写道：
> Hi, Rui Fan
> I'll start a discussion later.
>
> Best
> Chunjin Mu
>
> Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
>
> > Hi guys,
> >
> > I agree with monster. And I suggest use variable name, variable value and
> > description.
> >
> > By the way, there are some problems still need to confirm. What’s the
> > placeholder? And how to use them when edit sql or datastream job? It’s
> > better to discuss these asap.
> >
> > Best
> > fanrui
> >
> > On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <ma...@gmail.com> wrote:
> >
> > > hi, huajie, fan rui, monster
> > >
> > > Thank you for your suggestions.
> > >
> > > Now please Huajie to confirm how to do it. If remove the variable
> name, I
> > > can also accept it here. I will do small secondary development on our
> > > production system based on the community to meet internal needs.
> > >
> > > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五 15:58写道：
> > >
> > > > "外部采集整合kafka" is the Chinese description of variable, so it makes
> sense
> > > to
> > > > put it under `Description`.
> > > >
> > > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
> > > >
> > > > > The variable management module has three properties, including
> > variable
> > > > > code, variable name, and description, For example, the variable
> code
> > is
> > > > > "external.kafka.collect.brokers", and the variable name is
> > > "外部采集整合kafka".
> > > > > When we created the application, I forgot what the variable code
> was,
> > > > but I
> > > > > knew it was called "采集整合", so I can search Enter the "采集整合"
> keywords
> > in
> > > > the
> > > > > box, and of course you can also search for kafka keywords.
> > > > >
> > > > > At the same time, I want to know more detailed information about
> this
> > > > > kafka, such as who the operator is, what is the data retention
> > period,
> > > > and
> > > > > how big is the cluster size, but I can't use the description to
> > search,
> > > > so
> > > > > I can only search or variable code or variable name, and the
> variable
> > > > name
> > > > > is also for a simple display.
> > > > >
> > > > > Although external.kafka.collect.brokers already contains a lot of
> > > > > information, I think this is just a naming convention. When we
> > discuss
> > > > this
> > > > > variable, we will discuss it by using the "采集整合 kafka"
> > > > >
> > > > > We can discuss whether there is a problem with this design, what do
> > you
> > > > > think?
> > > > >
> > > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
> > > > >
> > > > > > Hi huajie:
> > > > > >
> > > > > > > If it is manually entered, it will always be treated as
> ordinary
> > > > text.
> > > > > >
> > > > > > Users always copy some code from an old job to a new job.
> > > > > > I'm afraid it might not meet the user's expectations.
> > > > > >
> > > > > > Best,
> > > > > > fanrui
> > > > > >
> > > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <be...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > hi 田向阳:
> > > > > > >
> > > > > > > The current discussion is about variable management, which is a
> > > > > > relatively
> > > > > > > independent function, No other modules involved, you can check
> > the
> > > > > thread
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > > > [1] thread:
> > > > > > >
> https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Huajie Wang
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > > > > > >
> > > > > > > > is this module include a resource management cente r where
> jars
> > > can
> > > > > be
> > > > > > > > chosen to provide to create a job
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > ---- 回复的原邮件 ----
> > > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > > > > > > | 日期 | 2022年10月08日 12:07 |
> > > > > > > > | 收件人 | dev@streampark.apache.org<de...@streampark.apache.org>
> |
> > > > > > > > | 抄送至 | |
> > > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management
> > module |
> > > > > > > > hi 阿穆:
> > > > > > > > Thank you for the clarification, My question is answered, but
> > > now I
> > > > > > have
> > > > > > > > one more concern,
> > > > > > > > I'm currently using {var} for variables, and I'm worried that
> > one
> > > > of
> > > > > > the
> > > > > > > > user's parameters includes "{" and "}", e.g :
> > > > > > > >
> > > > > > > > --password 'mypass{123}abc'
> > > > > > > >
> > > > > > > > So how do we parse parse this? How about ${var}
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Huajie Wang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> > > > > > > >
> > > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > > > > > > >
> > > > > > > > > Thanks for your reply.
> > > > > > > > >
> > > > > > > > > I have integrated your replies. I really don't pay much
> > > attention
> > > > > to
> > > > > > > SQL,
> > > > > > > > > so I will do the following.
> > > > > > > > >
> > > > > > > > > 1.Each team has independent variables that do not affect
> each
> > > > other
> > > > > > and
> > > > > > > > > are isolated from each other, but the same variable is
> unique
> > > > under
> > > > > > the
> > > > > > > > > same team, so you need to specify the team when creating
> > > > variables
> > > > > > > (users
> > > > > > > > > may belong to multiple teams, and the teams of the current
> > user
> > > > > will
> > > > > > be
> > > > > > > > > displayed here for users to choose)
> > > > > > > > > 2.For Flink jar jobs, there are two ways to pass variables
> > > > > > > > > - By using placeholders, such as adding --kafka {mykafka}
> to
> > > the
> > > > > args
> > > > > > > of
> > > > > > > > > the application, 'mykafka' is the variable code of a
> variable
> > > > > > > > > - Search by variable code or variable name. After a
> variable
> > is
> > > > > > > selected,
> > > > > > > > > the parameter transfer method is displayed
> > > > > > > > > 3.Flink sql will use variables as placeholders, E.g:
> > > > > > > > >
> > > > > > > > > CREATE TABLE KafkaTableA (
> > > > > > > > > `user_id` BIGINT,
> > > > > > > > > `item_id` BIGINT,
> > > > > > > > > `type` STRING
> > > > > > > > > ) WITH (
> > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > 'topic' = 'user_behaviorA',
> > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > '{kafka_clusterA}',
> > > > > > > > --
> > > > > > > > > 'kafka_ ClusterA' is the variable code
> > > > > > > > > 'properties.group.id' = 'testGroupA',
> > > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > > >
> > > > > > > > > CREATE TABLE KafkaTableB (
> > > > > > > > > `user_id` BIGINT,
> > > > > > > > > `item_id` BIGINT
> > > > > > > > > ) WITH (
> > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > 'topic' = 'user_behaviorB',
> > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > '{kafka_clusterB}',
> > > > > > > > --
> > > > > > > > > 'kafka_ ClusterB' is the variable code
> > > > > > > > > 'properties.group.id' = 'testGroupB',
> > > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > > >
> > > > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}'
> > FROM
> > > > > > > > > KafkaTableB -- 'type' is the variable code
> > > > > > > > >
> > > > > > > > > 4.In the future, we can consider setting some system
> built-in
> > > > > > variables
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > ChunJin Mu
> > > > > > > > >
> > > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <
> 1996fanrui@gmail.com
> > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi 阿穆:
> > > > > > > > > >
> > > > > > > > > > It's a great proposal.
> > > > > > > > > >
> > > > > > > > > > For usage scenarios, I strongly recommend support for
> > > > > replacing
> > > > > > > > > variables
> > > > > > > > > > in sql. E.g:
> > > > > > > > > >
> > > > > > > > > > CREATE TABLE KafkaTable (
> > > > > > > > > > `user_id` BIGINT,
> > > > > > > > > > `item_id` BIGINT
> > > > > > > > > > ) WITH (
> > > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > > 'topic' = 'user_behavior',
> > > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > > > '{kafka_clusterA}',
> > > > > > > > > > 'properties.group.id' =
> 'testGroup',
> > > > > > > > > > 'scan.startup.mode' =
> > 'earliest-offset',
> > > > > > > > > > 'format' = 'csv'
> > > > > > > > > > )
> > > > > > > > > >
> > > > > > > > > > Note: 'properties.bootstrap.servers' =
> > '{kafka_clusterA}',
> > > > > > > > > >
> > > > > > > > > > For variable management, I have some question:
> > > > > > > > > >
> > > > > > > > > > 1. Is the parameter used within the team? If all teams
> > > use a
> > > > > > > common
> > > > > > > > > > parameter, it
> > > > > > > > > > needs to be defined for each team, right?
> > > > > > > > > > 2. Which users can modify variables? Can all users
> > modify
> > > > all
> > > > > > > > > variables of
> > > > > > > > > > their team?
> > > > > > > > > > After the variable is modified incorrectly, many jobs
> > may
> > > be
> > > > > > > > > affected. So I
> > > > > > > > > > suggest that
> > > > > > > > > > only team admin or system admin have permission to
> > modify,
> > > > add
> > > > > > and
> > > > > > > > > delete
> > > > > > > > > > variables.
> > > > > > > > > >
> > > > > > > > > > Best
> > > > > > > > > > fanrui
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <
> > benjobx@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > hi 阿穆:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks for starting a very useful discussion, I have a
> > > some
> > > > > > > > questions:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1. The access boundary problem of variable management,
> > is
> > > > this
> > > > > > all
> > > > > > > > > > people in the platform can access these variables( no
> > > matter
> > > > > who
> > > > > > > > > > defines) or that only team members can be access these
> > > > > > variables.
> > > > > > > > > >
> > > > > > > > > > 2. There are two usage scenarios for variables that I
> > know
> > > > of,
> > > > > > one
> > > > > > > > is
> > > > > > > > > > in the entry parameter of main, and the other is in
> > > > flinksql.
> > > > > > Are
> > > > > > > > both
> > > > > > > > > > supported, or only supported main entry ?
> > > > > > > > > >
> > > > > > > > > > about first question, my opinion:
> > > > > > > > > >
> > > > > > > > > > 1. The variable management is associated with the
> team.
> > > > After
> > > > > > the
> > > > > > > > user
> > > > > > > > > > logs in to the system, he must select a team (a user
> can
> > > > > belong
> > > > > > to
> > > > > > > > one
> > > > > > > > > > or more teams), The variable belongs to a team
> > > > > > > > > >
> > > > > > > > > > 2. All resources between teams are isolated, which
> means
> > > > that
> > > > > > > > > > variables are also isolated. In short, members of a
> team
> > > can
> > > > > use
> > > > > > > and
> > > > > > > > > > define these variables, which are invisible to other
> > > > members,
> > > > > > and
> > > > > > > > > > other members cannot use these variables.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Welcome to your views
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Huajie Wang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com> 于2022年10月7日周五
> > > > 14:18写道：
> > > > > > > > >
> > > > > > > > > > > Is it possible to do this variable support in sql
> > as
> > > > well
> > > > > > > > > > >
> > > > > > > > > > > 阿穆 <305343894@qq.com.invalid> 于2022年10月7日周五
> > > > 13:54写道：
> > > > > > > > > > >
> > > > > > > > > > > > Hi guys,
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > I want to start a discussion about the
> > variable
> > > > > > > management
> > > > > > > > > module. Please
> > > > > > > > > > > > see this issue issue-1477.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Background:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > In the actual production environment, Flink
> > jobs
> > > > are
> > > > > > > > > generally complex,
> > > > > > > > > > > > and usually require multiple external
> > > components.
> > > > > For
> > > > > > > > > example, Flink jobs
> > > > > > > > > > > > consume data from Kafka, then connect
> external
> > > > > > > components
> > > > > > > > > such as HBase
> > > > > > > > > > > or
> > > > > > > > > > > > Redis to obtain additional business
> > information,
> > > > and
> > > > > > > then
> > > > > > > > > write it to the
> > > > > > > > > > > > downstream external components. There are
> the
> > > > > > following
> > > > > > > > > problems:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 1. The connection information of external
> > > > > components,
> > > > > > > such
> > > > > > > > > as IP, port
> > > > > > > > > > > and
> > > > > > > > > > > > user password, needs to be configured in the
> > > > > > application
> > > > > > > > > args and
> > > > > > > > > > > > transferred to the Flink job, so that the
> > > > connection
> > > > > > > > > information of
> > > > > > > > > > > > external components is distributed in
> multiple
> > > > > > > > > applications. Once the
> > > > > > > > > > > > connection information of external
> components
> > > > > changes,
> > > > > > > > many
> > > > > > > > > application
> > > > > > > > > > > > args parameters need to be modified, which
> > will
> > > > lead
> > > > > > to
> > > > > > > > > high operation
> > > > > > > > > > > and
> > > > > > > > > > > > maintenance costs.
> > > > > > > > > > > > 2. There are many people in the team
> > developing
> > > > > > > real-time
> > > > > > > > > computing jobs.
> > > > > > > > > > > > There is no uniform specification for the
> > > > connection
> > > > > > > > > information of the
> > > > > > > > > > > > external components passed to the job,
> > resulting
> > > > in
> > > > > > > > > different parameter
> > > > > > > > > > > > names of the same component. This is
> difficult
> > > to
> > > > > > count
> > > > > > > > > which external
> > > > > > > > > > > > components are dependent.
> > > > > > > > > > > > 3. In production practice, there are usually
> > > > > multiple
> > > > > > > sets
> > > > > > > > > of
> > > > > > > > > > > > environments, such as test environment and
> > > > > production
> > > > > > > > > environment. It is
> > > > > > > > > > > > not intuitive to judge whether a component
> > > belongs
> > > > > to
> > > > > > a
> > > > > > > > > test environment
> > > > > > > > > > > or
> > > > > > > > > > > > a production environment simply through IP
> and
> > > > > ports.
> > > > > > > Even
> > > > > > > > > if it can be
> > > > > > > > > > > > judged, there are some omissions. In this
> way,
> > > the
> > > > > > > > > connection information
> > > > > > > > > > > > online to the production environment may be
> > > > external
> > > > > > > > > components of the
> > > > > > > > > > > test
> > > > > > > > > > > > environment, or components of the production
> > > > > > environment
> > > > > > > > are
> > > > > > > > > > > inadvertently
> > > > > > > > > > > > configured during testing, leading to
> > production
> > > > > > > failures.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Proposal:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 1. Create a unified module to manage
> > variables.
> > > > > Users
> > > > > > > can
> > > > > > > > > define their
> > > > > > > > > > > own
> > > > > > > > > > > > variables in this module. Variable
> information
> > > > > > includes
> > > > > > > > > variable code
> > > > > > > > > > > > (passed to Flink program as parameter name),
> > > > > variable
> > > > > > > > > value, variable
> > > > > > > > > > name,
> > > > > > > > > > > > variable description, and which applications
> > > > depend
> > > > > on
> > > > > > > > > > > > 2. In the same Team module, the variable
> code
> > or
> > > > > > > variable
> > > > > > > > > name is unique
> > > > > > > > > > > > - By using placeholders, such as adding --
> > kafka
> > > > > > > {mykafka}
> > > > > > > > > to the args of
> > > > > > > > > > > > the application, mykafka is the variable
> code
> > > of a
> > > > > > > > variable
> > > > > > > > > > > > - Search by variable code or variable name.
> > > After
> > > > a
> > > > > > > > > variable is selected,
> > > > > > > > > > > > the parameter transfer method is displayed
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > If you have any suggestions, welcome to
> > > > participate
> > > > > in
> > > > > > > the
> > > > > > > > > discussion on
> > > > > > > > > > > > the mail list or issue, I look forward to
> your
> > > > > > feedback.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Best wishes
> > > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Chunjin Mu <ma...@gmail.com>.
Hi, Rui Fan
I'll start a discussion later.
Best
Chunjin Mu
Rui Fan <19...@gmail.com> 于2022年10月14日周五 19:42写道：
> Hi guys,
>
> I agree with monster. And I suggest use variable name, variable value and
> description.
>
> By the way, there are some problems still need to confirm. What’s the
> placeholder? And how to use them when edit sql or datastream job? It’s
> better to discuss these asap.
>
> Best
> fanrui
>
> On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <ma...@gmail.com> wrote:
>
> > hi, huajie, fan rui, monster
> >
> > Thank you for your suggestions.
> >
> > Now please Huajie to confirm how to do it. If remove the variable name, I
> > can also accept it here. I will do small secondary development on our
> > production system based on the community to meet internal needs.
> >
> > 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五 15:58写道：
> >
> > > "外部采集整合kafka" is the Chinese description of variable, so it makes sense
> > to
> > > put it under `Description`.
> > >
> > > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
> > >
> > > > The variable management module has three properties, including
> variable
> > > > code, variable name, and description, For example, the variable code
> is
> > > > "external.kafka.collect.brokers", and the variable name is
> > "外部采集整合kafka".
> > > > When we created the application, I forgot what the variable code was,
> > > but I
> > > > knew it was called "采集整合", so I can search Enter the "采集整合" keywords
> in
> > > the
> > > > box, and of course you can also search for kafka keywords.
> > > >
> > > > At the same time, I want to know more detailed information about this
> > > > kafka, such as who the operator is, what is the data retention
> period,
> > > and
> > > > how big is the cluster size, but I can't use the description to
> search,
> > > so
> > > > I can only search or variable code or variable name, and the variable
> > > name
> > > > is also for a simple display.
> > > >
> > > > Although external.kafka.collect.brokers already contains a lot of
> > > > information, I think this is just a naming convention. When we
> discuss
> > > this
> > > > variable, we will discuss it by using the "采集整合 kafka"
> > > >
> > > > We can discuss whether there is a problem with this design, what do
> you
> > > > think?
> > > >
> > > > Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
> > > >
> > > > > Hi huajie:
> > > > >
> > > > > > If it is manually entered, it will always be treated as ordinary
> > > text.
> > > > >
> > > > > Users always copy some code from an old job to a new job.
> > > > > I'm afraid it might not meet the user's expectations.
> > > > >
> > > > > Best,
> > > > > fanrui
> > > > >
> > > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <be...@gmail.com>
> > wrote:
> > > > >
> > > > > > hi 田向阳:
> > > > > >
> > > > > > The current discussion is about variable management, which is a
> > > > > relatively
> > > > > > independent function, No other modules involved, you can check
> the
> > > > thread
> > > > > > [1]
> > > > > >
> > > > > >
> > > > > > [1] thread:
> > > > > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Huajie Wang
> > > > > >
> > > > > >
> > > > > >
> > > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > > > > >
> > > > > > > is this module include a resource management cente r where jars
> > can
> > > > be
> > > > > > > chosen to provide to create a job
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ---- 回复的原邮件 ----
> > > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > > > > > | 日期 | 2022年10月08日 12:07 |
> > > > > > > | 收件人 | dev@streampark.apache.org<de...@streampark.apache.org> |
> > > > > > > | 抄送至 | |
> > > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management
> module |
> > > > > > > hi 阿穆:
> > > > > > > Thank you for the clarification, My question is answered, but
> > now I
> > > > > have
> > > > > > > one more concern,
> > > > > > > I'm currently using {var} for variables, and I'm worried that
> one
> > > of
> > > > > the
> > > > > > > user's parameters includes "{" and "}", e.g :
> > > > > > >
> > > > > > > --password 'mypass{123}abc'
> > > > > > >
> > > > > > > So how do we parse parse this? How about ${var}
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Huajie Wang
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> > > > > > >
> > > > > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > > > > > >
> > > > > > > > Thanks for your reply.
> > > > > > > >
> > > > > > > > I have integrated your replies. I really don't pay much
> > attention
> > > > to
> > > > > > SQL,
> > > > > > > > so I will do the following.
> > > > > > > >
> > > > > > > > 1.Each team has independent variables that do not affect each
> > > other
> > > > > and
> > > > > > > > are isolated from each other, but the same variable is unique
> > > under
> > > > > the
> > > > > > > > same team, so you need to specify the team when creating
> > > variables
> > > > > > (users
> > > > > > > > may belong to multiple teams, and the teams of the current
> user
> > > > will
> > > > > be
> > > > > > > > displayed here for users to choose)
> > > > > > > > 2.For Flink jar jobs, there are two ways to pass variables
> > > > > > > > - By using placeholders, such as adding --kafka {mykafka} to
> > the
> > > > args
> > > > > > of
> > > > > > > > the application, 'mykafka' is the variable code of a variable
> > > > > > > > - Search by variable code or variable name. After a variable
> is
> > > > > > selected,
> > > > > > > > the parameter transfer method is displayed
> > > > > > > > 3.Flink sql will use variables as placeholders, E.g:
> > > > > > > >
> > > > > > > > CREATE TABLE KafkaTableA (
> > > > > > > > `user_id` BIGINT,
> > > > > > > > `item_id` BIGINT,
> > > > > > > > `type` STRING
> > > > > > > > ) WITH (
> > > > > > > > 'connector' = 'kafka',
> > > > > > > > 'topic' = 'user_behaviorA',
> > > > > > > > 'properties.bootstrap.servers' =
> > > > > '{kafka_clusterA}',
> > > > > > > --
> > > > > > > > 'kafka_ ClusterA' is the variable code
> > > > > > > > 'properties.group.id' = 'testGroupA',
> > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > >
> > > > > > > > CREATE TABLE KafkaTableB (
> > > > > > > > `user_id` BIGINT,
> > > > > > > > `item_id` BIGINT
> > > > > > > > ) WITH (
> > > > > > > > 'connector' = 'kafka',
> > > > > > > > 'topic' = 'user_behaviorB',
> > > > > > > > 'properties.bootstrap.servers' =
> > > > > '{kafka_clusterB}',
> > > > > > > --
> > > > > > > > 'kafka_ ClusterB' is the variable code
> > > > > > > > 'properties.group.id' = 'testGroupB',
> > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > >
> > > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}'
> FROM
> > > > > > > > KafkaTableB -- 'type' is the variable code
> > > > > > > >
> > > > > > > > 4.In the future, we can consider setting some system built-in
> > > > > variables
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > ChunJin Mu
> > > > > > > >
> > > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <1996fanrui@gmail.com
> > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi 阿穆:
> > > > > > > > >
> > > > > > > > > It's a great proposal.
> > > > > > > > >
> > > > > > > > > For usage scenarios, I strongly recommend support for
> > > > replacing
> > > > > > > > variables
> > > > > > > > > in sql. E.g:
> > > > > > > > >
> > > > > > > > > CREATE TABLE KafkaTable (
> > > > > > > > > `user_id` BIGINT,
> > > > > > > > > `item_id` BIGINT
> > > > > > > > > ) WITH (
> > > > > > > > > 'connector' = 'kafka',
> > > > > > > > > 'topic' = 'user_behavior',
> > > > > > > > > 'properties.bootstrap.servers' =
> > > > > > > '{kafka_clusterA}',
> > > > > > > > > 'properties.group.id' = 'testGroup',
> > > > > > > > > 'scan.startup.mode' =
> 'earliest-offset',
> > > > > > > > > 'format' = 'csv'
> > > > > > > > > )
> > > > > > > > >
> > > > > > > > > Note: 'properties.bootstrap.servers' =
> '{kafka_clusterA}',
> > > > > > > > >
> > > > > > > > > For variable management, I have some question:
> > > > > > > > >
> > > > > > > > > 1. Is the parameter used within the team? If all teams
> > use a
> > > > > > common
> > > > > > > > > parameter, it
> > > > > > > > > needs to be defined for each team, right?
> > > > > > > > > 2. Which users can modify variables? Can all users
> modify
> > > all
> > > > > > > > variables of
> > > > > > > > > their team?
> > > > > > > > > After the variable is modified incorrectly, many jobs
> may
> > be
> > > > > > > > affected. So I
> > > > > > > > > suggest that
> > > > > > > > > only team admin or system admin have permission to
> modify,
> > > add
> > > > > and
> > > > > > > > delete
> > > > > > > > > variables.
> > > > > > > > >
> > > > > > > > > Best
> > > > > > > > > fanrui
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <
> benjobx@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > hi 阿穆:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks for starting a very useful discussion, I have a
> > some
> > > > > > > questions:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1. The access boundary problem of variable management,
> is
> > > this
> > > > > all
> > > > > > > > > people in the platform can access these variables( no
> > matter
> > > > who
> > > > > > > > > defines) or that only team members can be access these
> > > > > variables.
> > > > > > > > >
> > > > > > > > > 2. There are two usage scenarios for variables that I
> know
> > > of,
> > > > > one
> > > > > > > is
> > > > > > > > > in the entry parameter of main, and the other is in
> > > flinksql.
> > > > > Are
> > > > > > > both
> > > > > > > > > supported, or only supported main entry ?
> > > > > > > > >
> > > > > > > > > about first question, my opinion:
> > > > > > > > >
> > > > > > > > > 1. The variable management is associated with the team.
> > > After
> > > > > the
> > > > > > > user
> > > > > > > > > logs in to the system, he must select a team (a user can
> > > > belong
> > > > > to
> > > > > > > one
> > > > > > > > > or more teams), The variable belongs to a team
> > > > > > > > >
> > > > > > > > > 2. All resources between teams are isolated, which means
> > > that
> > > > > > > > > variables are also isolated. In short, members of a team
> > can
> > > > use
> > > > > > and
> > > > > > > > > define these variables, which are invisible to other
> > > members,
> > > > > and
> > > > > > > > > other members cannot use these variables.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Welcome to your views
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Huajie Wang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com> 于2022年10月7日周五
> > > 14:18写道：
> > > > > > > >
> > > > > > > > > > Is it possible to do this variable support in sql
> as
> > > well
> > > > > > > > > >
> > > > > > > > > > 阿穆 <305343894@qq.com.invalid> 于2022年10月7日周五
> > > 13:54写道：
> > > > > > > > > >
> > > > > > > > > > > Hi guys,
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I want to start a discussion about the
> variable
> > > > > > management
> > > > > > > > module. Please
> > > > > > > > > > > see this issue issue-1477.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Background:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > In the actual production environment, Flink
> jobs
> > > are
> > > > > > > > generally complex,
> > > > > > > > > > > and usually require multiple external
> > components.
> > > > For
> > > > > > > > example, Flink jobs
> > > > > > > > > > > consume data from Kafka, then connect external
> > > > > > components
> > > > > > > > such as HBase
> > > > > > > > > > or
> > > > > > > > > > > Redis to obtain additional business
> information,
> > > and
> > > > > > then
> > > > > > > > write it to the
> > > > > > > > > > > downstream external components. There are the
> > > > > following
> > > > > > > > problems:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 1. The connection information of external
> > > > components,
> > > > > > such
> > > > > > > > as IP, port
> > > > > > > > > > and
> > > > > > > > > > > user password, needs to be configured in the
> > > > > application
> > > > > > > > args and
> > > > > > > > > > > transferred to the Flink job, so that the
> > > connection
> > > > > > > > information of
> > > > > > > > > > > external components is distributed in multiple
> > > > > > > > applications. Once the
> > > > > > > > > > > connection information of external components
> > > > changes,
> > > > > > > many
> > > > > > > > application
> > > > > > > > > > > args parameters need to be modified, which
> will
> > > lead
> > > > > to
> > > > > > > > high operation
> > > > > > > > > > and
> > > > > > > > > > > maintenance costs.
> > > > > > > > > > > 2. There are many people in the team
> developing
> > > > > > real-time
> > > > > > > > computing jobs.
> > > > > > > > > > > There is no uniform specification for the
> > > connection
> > > > > > > > information of the
> > > > > > > > > > > external components passed to the job,
> resulting
> > > in
> > > > > > > > different parameter
> > > > > > > > > > > names of the same component. This is difficult
> > to
> > > > > count
> > > > > > > > which external
> > > > > > > > > > > components are dependent.
> > > > > > > > > > > 3. In production practice, there are usually
> > > > multiple
> > > > > > sets
> > > > > > > > of
> > > > > > > > > > > environments, such as test environment and
> > > > production
> > > > > > > > environment. It is
> > > > > > > > > > > not intuitive to judge whether a component
> > belongs
> > > > to
> > > > > a
> > > > > > > > test environment
> > > > > > > > > > or
> > > > > > > > > > > a production environment simply through IP and
> > > > ports.
> > > > > > Even
> > > > > > > > if it can be
> > > > > > > > > > > judged, there are some omissions. In this way,
> > the
> > > > > > > > connection information
> > > > > > > > > > > online to the production environment may be
> > > external
> > > > > > > > components of the
> > > > > > > > > > test
> > > > > > > > > > > environment, or components of the production
> > > > > environment
> > > > > > > are
> > > > > > > > > > inadvertently
> > > > > > > > > > > configured during testing, leading to
> production
> > > > > > failures.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Proposal:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 1. Create a unified module to manage
> variables.
> > > > Users
> > > > > > can
> > > > > > > > define their
> > > > > > > > > > own
> > > > > > > > > > > variables in this module. Variable information
> > > > > includes
> > > > > > > > variable code
> > > > > > > > > > > (passed to Flink program as parameter name),
> > > > variable
> > > > > > > > value, variable
> > > > > > > > > name,
> > > > > > > > > > > variable description, and which applications
> > > depend
> > > > on
> > > > > > > > > > > 2. In the same Team module, the variable code
> or
> > > > > > variable
> > > > > > > > name is unique
> > > > > > > > > > > - By using placeholders, such as adding --
> kafka
> > > > > > {mykafka}
> > > > > > > > to the args of
> > > > > > > > > > > the application, mykafka is the variable code
> > of a
> > > > > > > variable
> > > > > > > > > > > - Search by variable code or variable name.
> > After
> > > a
> > > > > > > > variable is selected,
> > > > > > > > > > > the parameter transfer method is displayed
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > If you have any suggestions, welcome to
> > > participate
> > > > in
> > > > > > the
> > > > > > > > discussion on
> > > > > > > > > > > the mail list or issue, I look forward to your
> > > > > feedback.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Best wishes
> > > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Rui Fan <19...@gmail.com>.
Hi guys,
I agree with monster. And I suggest use variable name, variable value and
description.
By the way, there are some problems still need to confirm. What’s the
placeholder? And how to use them when edit sql or datastream job? It’s
better to discuss these asap.
Best
fanrui
On Fri, Oct 14, 2022 at 17:44 Chunjin Mu <ma...@gmail.com> wrote:
> hi, huajie, fan rui, monster
>
> Thank you for your suggestions.
>
> Now please Huajie to confirm how to do it. If remove the variable name, I
> can also accept it here. I will do small secondary development on our
> production system based on the community to meet internal needs.
>
> 陈卓宇 <mo...@gmail.com> 于2022年10月14日周五 15:58写道：
>
> > "外部采集整合kafka" is the Chinese description of variable, so it makes sense
> to
> > put it under `Description`.
> >
> > Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
> >
> > > The variable management module has three properties, including variable
> > > code, variable name, and description, For example, the variable code is
> > > "external.kafka.collect.brokers", and the variable name is
> "外部采集整合kafka".
> > > When we created the application, I forgot what the variable code was,
> > but I
> > > knew it was called "采集整合", so I can search Enter the "采集整合" keywords in
> > the
> > > box, and of course you can also search for kafka keywords.
> > >
> > > At the same time, I want to know more detailed information about this
> > > kafka, such as who the operator is, what is the data retention period,
> > and
> > > how big is the cluster size, but I can't use the description to search,
> > so
> > > I can only search or variable code or variable name, and the variable
> > name
> > > is also for a simple display.
> > >
> > > Although external.kafka.collect.brokers already contains a lot of
> > > information, I think this is just a naming convention. When we discuss
> > this
> > > variable, we will discuss it by using the "采集整合 kafka"
> > >
> > > We can discuss whether there is a problem with this design, what do you
> > > think?
> > >
> > > Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
> > >
> > > > Hi huajie:
> > > >
> > > > > If it is manually entered, it will always be treated as ordinary
> > text.
> > > >
> > > > Users always copy some code from an old job to a new job.
> > > > I'm afraid it might not meet the user's expectations.
> > > >
> > > > Best,
> > > > fanrui
> > > >
> > > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <be...@gmail.com>
> wrote:
> > > >
> > > > > hi 田向阳:
> > > > >
> > > > > The current discussion is about variable management, which is a
> > > > relatively
> > > > > independent function, No other modules involved, you can check the
> > > thread
> > > > > [1]
> > > > >
> > > > >
> > > > > [1] thread:
> > > > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > > > >
> > > > >
> > > > > Best,
> > > > > Huajie Wang
> > > > >
> > > > >
> > > > >
> > > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > > > >
> > > > > > is this module include a resource management cente r where jars
> can
> > > be
> > > > > > chosen to provide to create a job
> > > > > >
> > > > > >
> > > > > >
> > > > > > ---- 回复的原邮件 ----
> > > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > > > > | 日期 | 2022年10月08日 12:07 |
> > > > > > | 收件人 | dev@streampark.apache.org<de...@streampark.apache.org> |
> > > > > > | 抄送至 | |
> > > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management module |
> > > > > > hi 阿穆:
> > > > > > Thank you for the clarification, My question is answered, but
> now I
> > > > have
> > > > > > one more concern,
> > > > > > I'm currently using {var} for variables, and I'm worried that one
> > of
> > > > the
> > > > > > user's parameters includes "{" and "}", e.g :
> > > > > >
> > > > > > --password 'mypass{123}abc'
> > > > > >
> > > > > > So how do we parse parse this? How about ${var}
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Huajie Wang
> > > > > >
> > > > > >
> > > > > >
> > > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> > > > > >
> > > > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > > > > >
> > > > > > > Thanks for your reply.
> > > > > > >
> > > > > > > I have integrated your replies. I really don't pay much
> attention
> > > to
> > > > > SQL,
> > > > > > > so I will do the following.
> > > > > > >
> > > > > > > 1.Each team has independent variables that do not affect each
> > other
> > > > and
> > > > > > > are isolated from each other, but the same variable is unique
> > under
> > > > the
> > > > > > > same team, so you need to specify the team when creating
> > variables
> > > > > (users
> > > > > > > may belong to multiple teams, and the teams of the current user
> > > will
> > > > be
> > > > > > > displayed here for users to choose)
> > > > > > > 2.For Flink jar jobs, there are two ways to pass variables
> > > > > > > - By using placeholders, such as adding --kafka {mykafka} to
> the
> > > args
> > > > > of
> > > > > > > the application, 'mykafka' is the variable code of a variable
> > > > > > > - Search by variable code or variable name. After a variable is
> > > > > selected,
> > > > > > > the parameter transfer method is displayed
> > > > > > > 3.Flink sql will use variables as placeholders, E.g:
> > > > > > >
> > > > > > > CREATE TABLE KafkaTableA (
> > > > > > > `user_id` BIGINT,
> > > > > > > `item_id` BIGINT,
> > > > > > > `type` STRING
> > > > > > > ) WITH (
> > > > > > > 'connector' = 'kafka',
> > > > > > > 'topic' = 'user_behaviorA',
> > > > > > > 'properties.bootstrap.servers' =
> > > > '{kafka_clusterA}',
> > > > > > --
> > > > > > > 'kafka_ ClusterA' is the variable code
> > > > > > > 'properties.group.id' = 'testGroupA',
> > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > >
> > > > > > > CREATE TABLE KafkaTableB (
> > > > > > > `user_id` BIGINT,
> > > > > > > `item_id` BIGINT
> > > > > > > ) WITH (
> > > > > > > 'connector' = 'kafka',
> > > > > > > 'topic' = 'user_behaviorB',
> > > > > > > 'properties.bootstrap.servers' =
> > > > '{kafka_clusterB}',
> > > > > > --
> > > > > > > 'kafka_ ClusterB' is the variable code
> > > > > > > 'properties.group.id' = 'testGroupB',
> > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > >
> > > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM
> > > > > > > KafkaTableB -- 'type' is the variable code
> > > > > > >
> > > > > > > 4.In the future, we can consider setting some system built-in
> > > > variables
> > > > > > >
> > > > > > > Best,
> > > > > > > ChunJin Mu
> > > > > > >
> > > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <1996fanrui@gmail.com
> >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi 阿穆:
> > > > > > > >
> > > > > > > > It's a great proposal.
> > > > > > > >
> > > > > > > > For usage scenarios, I strongly recommend support for
> > > replacing
> > > > > > > variables
> > > > > > > > in sql. E.g:
> > > > > > > >
> > > > > > > > CREATE TABLE KafkaTable (
> > > > > > > > `user_id` BIGINT,
> > > > > > > > `item_id` BIGINT
> > > > > > > > ) WITH (
> > > > > > > > 'connector' = 'kafka',
> > > > > > > > 'topic' = 'user_behavior',
> > > > > > > > 'properties.bootstrap.servers' =
> > > > > > '{kafka_clusterA}',
> > > > > > > > 'properties.group.id' = 'testGroup',
> > > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > > 'format' = 'csv'
> > > > > > > > )
> > > > > > > >
> > > > > > > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > > > > > > >
> > > > > > > > For variable management, I have some question:
> > > > > > > >
> > > > > > > > 1. Is the parameter used within the team? If all teams
> use a
> > > > > common
> > > > > > > > parameter, it
> > > > > > > > needs to be defined for each team, right?
> > > > > > > > 2. Which users can modify variables? Can all users modify
> > all
> > > > > > > variables of
> > > > > > > > their team?
> > > > > > > > After the variable is modified incorrectly, many jobs may
> be
> > > > > > > affected. So I
> > > > > > > > suggest that
> > > > > > > > only team admin or system admin have permission to modify,
> > add
> > > > and
> > > > > > > delete
> > > > > > > > variables.
> > > > > > > >
> > > > > > > > Best
> > > > > > > > fanrui
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <benjobx@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > hi 阿穆:
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks for starting a very useful discussion, I have a
> some
> > > > > > questions:
> > > > > > > >
> > > > > > > >
> > > > > > > > 1. The access boundary problem of variable management, is
> > this
> > > > all
> > > > > > > > people in the platform can access these variables( no
> matter
> > > who
> > > > > > > > defines) or that only team members can be access these
> > > > variables.
> > > > > > > >
> > > > > > > > 2. There are two usage scenarios for variables that I know
> > of,
> > > > one
> > > > > > is
> > > > > > > > in the entry parameter of main, and the other is in
> > flinksql.
> > > > Are
> > > > > > both
> > > > > > > > supported, or only supported main entry ?
> > > > > > > >
> > > > > > > > about first question, my opinion:
> > > > > > > >
> > > > > > > > 1. The variable management is associated with the team.
> > After
> > > > the
> > > > > > user
> > > > > > > > logs in to the system, he must select a team (a user can
> > > belong
> > > > to
> > > > > > one
> > > > > > > > or more teams), The variable belongs to a team
> > > > > > > >
> > > > > > > > 2. All resources between teams are isolated, which means
> > that
> > > > > > > > variables are also isolated. In short, members of a team
> can
> > > use
> > > > > and
> > > > > > > > define these variables, which are invisible to other
> > members,
> > > > and
> > > > > > > > other members cannot use these variables.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Welcome to your views
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Huajie Wang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com> 于2022年10月7日周五
> > 14:18写道：
> > > > > > >
> > > > > > > > > Is it possible to do this variable support in sql as
> > well
> > > > > > > > >
> > > > > > > > > 阿穆 <305343894@qq.com.invalid> 于2022年10月7日周五
> > 13:54写道：
> > > > > > > > >
> > > > > > > > > > Hi guys,
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > I want to start a discussion about the variable
> > > > > management
> > > > > > > module. Please
> > > > > > > > > > see this issue issue-1477.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Background:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > In the actual production environment, Flink jobs
> > are
> > > > > > > generally complex,
> > > > > > > > > > and usually require multiple external
> components.
> > > For
> > > > > > > example, Flink jobs
> > > > > > > > > > consume data from Kafka, then connect external
> > > > > components
> > > > > > > such as HBase
> > > > > > > > > or
> > > > > > > > > > Redis to obtain additional business information,
> > and
> > > > > then
> > > > > > > write it to the
> > > > > > > > > > downstream external components. There are the
> > > > following
> > > > > > > problems:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1. The connection information of external
> > > components,
> > > > > such
> > > > > > > as IP, port
> > > > > > > > > and
> > > > > > > > > > user password, needs to be configured in the
> > > > application
> > > > > > > args and
> > > > > > > > > > transferred to the Flink job, so that the
> > connection
> > > > > > > information of
> > > > > > > > > > external components is distributed in multiple
> > > > > > > applications. Once the
> > > > > > > > > > connection information of external components
> > > changes,
> > > > > > many
> > > > > > > application
> > > > > > > > > > args parameters need to be modified, which will
> > lead
> > > > to
> > > > > > > high operation
> > > > > > > > > and
> > > > > > > > > > maintenance costs.
> > > > > > > > > > 2. There are many people in the team developing
> > > > > real-time
> > > > > > > computing jobs.
> > > > > > > > > > There is no uniform specification for the
> > connection
> > > > > > > information of the
> > > > > > > > > > external components passed to the job, resulting
> > in
> > > > > > > different parameter
> > > > > > > > > > names of the same component. This is difficult
> to
> > > > count
> > > > > > > which external
> > > > > > > > > > components are dependent.
> > > > > > > > > > 3. In production practice, there are usually
> > > multiple
> > > > > sets
> > > > > > > of
> > > > > > > > > > environments, such as test environment and
> > > production
> > > > > > > environment. It is
> > > > > > > > > > not intuitive to judge whether a component
> belongs
> > > to
> > > > a
> > > > > > > test environment
> > > > > > > > > or
> > > > > > > > > > a production environment simply through IP and
> > > ports.
> > > > > Even
> > > > > > > if it can be
> > > > > > > > > > judged, there are some omissions. In this way,
> the
> > > > > > > connection information
> > > > > > > > > > online to the production environment may be
> > external
> > > > > > > components of the
> > > > > > > > > test
> > > > > > > > > > environment, or components of the production
> > > > environment
> > > > > > are
> > > > > > > > > inadvertently
> > > > > > > > > > configured during testing, leading to production
> > > > > failures.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Proposal:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 1. Create a unified module to manage variables.
> > > Users
> > > > > can
> > > > > > > define their
> > > > > > > > > own
> > > > > > > > > > variables in this module. Variable information
> > > > includes
> > > > > > > variable code
> > > > > > > > > > (passed to Flink program as parameter name),
> > > variable
> > > > > > > value, variable
> > > > > > > > name,
> > > > > > > > > > variable description, and which applications
> > depend
> > > on
> > > > > > > > > > 2. In the same Team module, the variable code or
> > > > > variable
> > > > > > > name is unique
> > > > > > > > > > - By using placeholders, such as adding -- kafka
> > > > > {mykafka}
> > > > > > > to the args of
> > > > > > > > > > the application, mykafka is the variable code
> of a
> > > > > > variable
> > > > > > > > > > - Search by variable code or variable name.
> After
> > a
> > > > > > > variable is selected,
> > > > > > > > > > the parameter transfer method is displayed
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > If you have any suggestions, welcome to
> > participate
> > > in
> > > > > the
> > > > > > > discussion on
> > > > > > > > > > the mail list or issue, I look forward to your
> > > > feedback.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best wishes
> > > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Chunjin Mu <ma...@gmail.com>.
hi, huajie, fan rui, monster
Thank you for your suggestions.
Now please Huajie to confirm how to do it. If remove the variable name, I
can also accept it here. I will do small secondary development on our
production system based on the community to meet internal needs.
陈卓宇 <mo...@gmail.com> 于2022年10月14日周五 15:58写道：
> "外部采集整合kafka" is the Chinese description of variable, so it makes sense to
> put it under `Description`.
>
> Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
>
> > The variable management module has three properties, including variable
> > code, variable name, and description, For example, the variable code is
> > "external.kafka.collect.brokers", and the variable name is "外部采集整合kafka".
> > When we created the application, I forgot what the variable code was,
> but I
> > knew it was called "采集整合", so I can search Enter the "采集整合" keywords in
> the
> > box, and of course you can also search for kafka keywords.
> >
> > At the same time, I want to know more detailed information about this
> > kafka, such as who the operator is, what is the data retention period,
> and
> > how big is the cluster size, but I can't use the description to search,
> so
> > I can only search or variable code or variable name, and the variable
> name
> > is also for a simple display.
> >
> > Although external.kafka.collect.brokers already contains a lot of
> > information, I think this is just a naming convention. When we discuss
> this
> > variable, we will discuss it by using the "采集整合 kafka"
> >
> > We can discuss whether there is a problem with this design, what do you
> > think?
> >
> > Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
> >
> > > Hi huajie:
> > >
> > > > If it is manually entered, it will always be treated as ordinary
> text.
> > >
> > > Users always copy some code from an old job to a new job.
> > > I'm afraid it might not meet the user's expectations.
> > >
> > > Best,
> > > fanrui
> > >
> > > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <be...@gmail.com> wrote:
> > >
> > > > hi 田向阳:
> > > >
> > > > The current discussion is about variable management, which is a
> > > relatively
> > > > independent function, No other modules involved, you can check the
> > thread
> > > > [1]
> > > >
> > > >
> > > > [1] thread:
> > > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > > >
> > > >
> > > > Best,
> > > > Huajie Wang
> > > >
> > > >
> > > >
> > > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > > >
> > > > > is this module include a resource management cente r where jars can
> > be
> > > > > chosen to provide to create a job
> > > > >
> > > > >
> > > > >
> > > > > ---- 回复的原邮件 ----
> > > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > > > | 日期 | 2022年10月08日 12:07 |
> > > > > | 收件人 | dev@streampark.apache.org<de...@streampark.apache.org> |
> > > > > | 抄送至 | |
> > > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management module |
> > > > > hi 阿穆:
> > > > > Thank you for the clarification, My question is answered, but now I
> > > have
> > > > > one more concern,
> > > > > I'm currently using {var} for variables, and I'm worried that one
> of
> > > the
> > > > > user's parameters includes "{" and "}", e.g :
> > > > >
> > > > > --password 'mypass{123}abc'
> > > > >
> > > > > So how do we parse parse this? How about ${var}
> > > > >
> > > > >
> > > > >
> > > > > Best,
> > > > > Huajie Wang
> > > > >
> > > > >
> > > > >
> > > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> > > > >
> > > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > > > >
> > > > > > Thanks for your reply.
> > > > > >
> > > > > > I have integrated your replies. I really don't pay much attention
> > to
> > > > SQL,
> > > > > > so I will do the following.
> > > > > >
> > > > > > 1.Each team has independent variables that do not affect each
> other
> > > and
> > > > > > are isolated from each other, but the same variable is unique
> under
> > > the
> > > > > > same team, so you need to specify the team when creating
> variables
> > > > (users
> > > > > > may belong to multiple teams, and the teams of the current user
> > will
> > > be
> > > > > > displayed here for users to choose)
> > > > > > 2.For Flink jar jobs, there are two ways to pass variables
> > > > > > - By using placeholders, such as adding --kafka {mykafka} to the
> > args
> > > > of
> > > > > > the application, 'mykafka' is the variable code of a variable
> > > > > > - Search by variable code or variable name. After a variable is
> > > > selected,
> > > > > > the parameter transfer method is displayed
> > > > > > 3.Flink sql will use variables as placeholders, E.g:
> > > > > >
> > > > > > CREATE TABLE KafkaTableA (
> > > > > > `user_id` BIGINT,
> > > > > > `item_id` BIGINT,
> > > > > > `type` STRING
> > > > > > ) WITH (
> > > > > > 'connector' = 'kafka',
> > > > > > 'topic' = 'user_behaviorA',
> > > > > > 'properties.bootstrap.servers' =
> > > '{kafka_clusterA}',
> > > > > --
> > > > > > 'kafka_ ClusterA' is the variable code
> > > > > > 'properties.group.id' = 'testGroupA',
> > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > >
> > > > > > CREATE TABLE KafkaTableB (
> > > > > > `user_id` BIGINT,
> > > > > > `item_id` BIGINT
> > > > > > ) WITH (
> > > > > > 'connector' = 'kafka',
> > > > > > 'topic' = 'user_behaviorB',
> > > > > > 'properties.bootstrap.servers' =
> > > '{kafka_clusterB}',
> > > > > --
> > > > > > 'kafka_ ClusterB' is the variable code
> > > > > > 'properties.group.id' = 'testGroupB',
> > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > >
> > > > > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM
> > > > > > KafkaTableB -- 'type' is the variable code
> > > > > >
> > > > > > 4.In the future, we can consider setting some system built-in
> > > variables
> > > > > >
> > > > > > Best,
> > > > > > ChunJin Mu
> > > > > >
> > > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <1996fanrui@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi 阿穆:
> > > > > > >
> > > > > > > It's a great proposal.
> > > > > > >
> > > > > > > For usage scenarios, I strongly recommend support for
> > replacing
> > > > > > variables
> > > > > > > in sql. E.g:
> > > > > > >
> > > > > > > CREATE TABLE KafkaTable (
> > > > > > > `user_id` BIGINT,
> > > > > > > `item_id` BIGINT
> > > > > > > ) WITH (
> > > > > > > 'connector' = 'kafka',
> > > > > > > 'topic' = 'user_behavior',
> > > > > > > 'properties.bootstrap.servers' =
> > > > > '{kafka_clusterA}',
> > > > > > > 'properties.group.id' = 'testGroup',
> > > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > > 'format' = 'csv'
> > > > > > > )
> > > > > > >
> > > > > > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > > > > > >
> > > > > > > For variable management, I have some question:
> > > > > > >
> > > > > > > 1. Is the parameter used within the team? If all teams use a
> > > > common
> > > > > > > parameter, it
> > > > > > > needs to be defined for each team, right?
> > > > > > > 2. Which users can modify variables? Can all users modify
> all
> > > > > > variables of
> > > > > > > their team?
> > > > > > > After the variable is modified incorrectly, many jobs may be
> > > > > > affected. So I
> > > > > > > suggest that
> > > > > > > only team admin or system admin have permission to modify,
> add
> > > and
> > > > > > delete
> > > > > > > variables.
> > > > > > >
> > > > > > > Best
> > > > > > > fanrui
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <benjobx@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > hi 阿穆:
> > > > > > >
> > > > > > >
> > > > > > > Thanks for starting a very useful discussion, I have a some
> > > > > questions:
> > > > > > >
> > > > > > >
> > > > > > > 1. The access boundary problem of variable management, is
> this
> > > all
> > > > > > > people in the platform can access these variables( no matter
> > who
> > > > > > > defines) or that only team members can be access these
> > > variables.
> > > > > > >
> > > > > > > 2. There are two usage scenarios for variables that I know
> of,
> > > one
> > > > > is
> > > > > > > in the entry parameter of main, and the other is in
> flinksql.
> > > Are
> > > > > both
> > > > > > > supported, or only supported main entry ?
> > > > > > >
> > > > > > > about first question, my opinion:
> > > > > > >
> > > > > > > 1. The variable management is associated with the team.
> After
> > > the
> > > > > user
> > > > > > > logs in to the system, he must select a team (a user can
> > belong
> > > to
> > > > > one
> > > > > > > or more teams), The variable belongs to a team
> > > > > > >
> > > > > > > 2. All resources between teams are isolated, which means
> that
> > > > > > > variables are also isolated. In short, members of a team can
> > use
> > > > and
> > > > > > > define these variables, which are invisible to other
> members,
> > > and
> > > > > > > other members cannot use these variables.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Welcome to your views
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Huajie Wang
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com> 于2022年10月7日周五
> 14:18写道：
> > > > > >
> > > > > > > > Is it possible to do this variable support in sql as
> well
> > > > > > > >
> > > > > > > > 阿穆 <305343894@qq.com.invalid> 于2022年10月7日周五
> 13:54写道：
> > > > > > > >
> > > > > > > > > Hi guys,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I want to start a discussion about the variable
> > > > management
> > > > > > module. Please
> > > > > > > > > see this issue issue-1477.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Background:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > In the actual production environment, Flink jobs
> are
> > > > > > generally complex,
> > > > > > > > > and usually require multiple external components.
> > For
> > > > > > example, Flink jobs
> > > > > > > > > consume data from Kafka, then connect external
> > > > components
> > > > > > such as HBase
> > > > > > > > or
> > > > > > > > > Redis to obtain additional business information,
> and
> > > > then
> > > > > > write it to the
> > > > > > > > > downstream external components. There are the
> > > following
> > > > > > problems:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1. The connection information of external
> > components,
> > > > such
> > > > > > as IP, port
> > > > > > > > and
> > > > > > > > > user password, needs to be configured in the
> > > application
> > > > > > args and
> > > > > > > > > transferred to the Flink job, so that the
> connection
> > > > > > information of
> > > > > > > > > external components is distributed in multiple
> > > > > > applications. Once the
> > > > > > > > > connection information of external components
> > changes,
> > > > > many
> > > > > > application
> > > > > > > > > args parameters need to be modified, which will
> lead
> > > to
> > > > > > high operation
> > > > > > > > and
> > > > > > > > > maintenance costs.
> > > > > > > > > 2. There are many people in the team developing
> > > > real-time
> > > > > > computing jobs.
> > > > > > > > > There is no uniform specification for the
> connection
> > > > > > information of the
> > > > > > > > > external components passed to the job, resulting
> in
> > > > > > different parameter
> > > > > > > > > names of the same component. This is difficult to
> > > count
> > > > > > which external
> > > > > > > > > components are dependent.
> > > > > > > > > 3. In production practice, there are usually
> > multiple
> > > > sets
> > > > > > of
> > > > > > > > > environments, such as test environment and
> > production
> > > > > > environment. It is
> > > > > > > > > not intuitive to judge whether a component belongs
> > to
> > > a
> > > > > > test environment
> > > > > > > > or
> > > > > > > > > a production environment simply through IP and
> > ports.
> > > > Even
> > > > > > if it can be
> > > > > > > > > judged, there are some omissions. In this way, the
> > > > > > connection information
> > > > > > > > > online to the production environment may be
> external
> > > > > > components of the
> > > > > > > > test
> > > > > > > > > environment, or components of the production
> > > environment
> > > > > are
> > > > > > > > inadvertently
> > > > > > > > > configured during testing, leading to production
> > > > failures.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Proposal:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 1. Create a unified module to manage variables.
> > Users
> > > > can
> > > > > > define their
> > > > > > > > own
> > > > > > > > > variables in this module. Variable information
> > > includes
> > > > > > variable code
> > > > > > > > > (passed to Flink program as parameter name),
> > variable
> > > > > > value, variable
> > > > > > > name,
> > > > > > > > > variable description, and which applications
> depend
> > on
> > > > > > > > > 2. In the same Team module, the variable code or
> > > > variable
> > > > > > name is unique
> > > > > > > > > - By using placeholders, such as adding -- kafka
> > > > {mykafka}
> > > > > > to the args of
> > > > > > > > > the application, mykafka is the variable code of a
> > > > > variable
> > > > > > > > > - Search by variable code or variable name. After
> a
> > > > > > variable is selected,
> > > > > > > > > the parameter transfer method is displayed
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > If you have any suggestions, welcome to
> participate
> > in
> > > > the
> > > > > > discussion on
> > > > > > > > > the mail list or issue, I look forward to your
> > > feedback.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best wishes
> > > > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by 陈卓宇 <mo...@gmail.com>.
"外部采集整合kafka" is the Chinese description of variable, so it makes sense to
put it under `Description`.
Chunjin Mu <ma...@gmail.com> 于2022年10月14日周五 12:01写道：
> The variable management module has three properties, including variable
> code, variable name, and description, For example, the variable code is
> "external.kafka.collect.brokers", and the variable name is "外部采集整合kafka".
> When we created the application, I forgot what the variable code was, but I
> knew it was called "采集整合", so I can search Enter the "采集整合" keywords in the
> box, and of course you can also search for kafka keywords.
>
> At the same time, I want to know more detailed information about this
> kafka, such as who the operator is, what is the data retention period, and
> how big is the cluster size, but I can't use the description to search, so
> I can only search or variable code or variable name, and the variable name
> is also for a simple display.
>
> Although external.kafka.collect.brokers already contains a lot of
> information, I think this is just a naming convention. When we discuss this
> variable, we will discuss it by using the "采集整合 kafka"
>
> We can discuss whether there is a problem with this design, what do you
> think?
>
> Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
>
> > Hi huajie:
> >
> > > If it is manually entered, it will always be treated as ordinary text.
> >
> > Users always copy some code from an old job to a new job.
> > I'm afraid it might not meet the user's expectations.
> >
> > Best,
> > fanrui
> >
> > On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <be...@gmail.com> wrote:
> >
> > > hi 田向阳:
> > >
> > > The current discussion is about variable management, which is a
> > relatively
> > > independent function, No other modules involved, you can check the
> thread
> > > [1]
> > >
> > >
> > > [1] thread:
> > > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> > >
> > >
> > > Best,
> > > Huajie Wang
> > >
> > >
> > >
> > > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> > >
> > > > is this module include a resource management cente r where jars can
> be
> > > > chosen to provide to create a job
> > > >
> > > >
> > > >
> > > > ---- 回复的原邮件 ----
> > > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > > | 日期 | 2022年10月08日 12:07 |
> > > > | 收件人 | dev@streampark.apache.org<de...@streampark.apache.org> |
> > > > | 抄送至 | |
> > > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management module |
> > > > hi 阿穆:
> > > > Thank you for the clarification, My question is answered, but now I
> > have
> > > > one more concern,
> > > > I'm currently using {var} for variables, and I'm worried that one of
> > the
> > > > user's parameters includes "{" and "}", e.g :
> > > >
> > > > --password 'mypass{123}abc'
> > > >
> > > > So how do we parse parse this? How about ${var}
> > > >
> > > >
> > > >
> > > > Best,
> > > > Huajie Wang
> > > >
> > > >
> > > >
> > > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> > > >
> > > > > Hi, monster, Huajie Wang, Rui Fan:
> > > > >
> > > > > Thanks for your reply.
> > > > >
> > > > > I have integrated your replies. I really don't pay much attention
> to
> > > SQL,
> > > > > so I will do the following.
> > > > >
> > > > > 1.Each team has independent variables that do not affect each other
> > and
> > > > > are isolated from each other, but the same variable is unique under
> > the
> > > > > same team, so you need to specify the team when creating variables
> > > (users
> > > > > may belong to multiple teams, and the teams of the current user
> will
> > be
> > > > > displayed here for users to choose)
> > > > > 2.For Flink jar jobs, there are two ways to pass variables
> > > > > - By using placeholders, such as adding --kafka {mykafka} to the
> args
> > > of
> > > > > the application, 'mykafka' is the variable code of a variable
> > > > > - Search by variable code or variable name. After a variable is
> > > selected,
> > > > > the parameter transfer method is displayed
> > > > > 3.Flink sql will use variables as placeholders, E.g:
> > > > >
> > > > > CREATE TABLE KafkaTableA (
> > > > > `user_id` BIGINT,
> > > > > `item_id` BIGINT,
> > > > > `type` STRING
> > > > > ) WITH (
> > > > > 'connector' = 'kafka',
> > > > > 'topic' = 'user_behaviorA',
> > > > > 'properties.bootstrap.servers' =
> > '{kafka_clusterA}',
> > > > --
> > > > > 'kafka_ ClusterA' is the variable code
> > > > > 'properties.group.id' = 'testGroupA',
> > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > >
> > > > > CREATE TABLE KafkaTableB (
> > > > > `user_id` BIGINT,
> > > > > `item_id` BIGINT
> > > > > ) WITH (
> > > > > 'connector' = 'kafka',
> > > > > 'topic' = 'user_behaviorB',
> > > > > 'properties.bootstrap.servers' =
> > '{kafka_clusterB}',
> > > > --
> > > > > 'kafka_ ClusterB' is the variable code
> > > > > 'properties.group.id' = 'testGroupB',
> > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > >
> > > > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM
> > > > > KafkaTableB -- 'type' is the variable code
> > > > >
> > > > > 4.In the future, we can consider setting some system built-in
> > variables
> > > > >
> > > > > Best,
> > > > > ChunJin Mu
> > > > >
> > > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <1996fanrui@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi 阿穆:
> > > > > >
> > > > > > It's a great proposal.
> > > > > >
> > > > > > For usage scenarios, I strongly recommend support for
> replacing
> > > > > variables
> > > > > > in sql. E.g:
> > > > > >
> > > > > > CREATE TABLE KafkaTable (
> > > > > > `user_id` BIGINT,
> > > > > > `item_id` BIGINT
> > > > > > ) WITH (
> > > > > > 'connector' = 'kafka',
> > > > > > 'topic' = 'user_behavior',
> > > > > > 'properties.bootstrap.servers' =
> > > > '{kafka_clusterA}',
> > > > > > 'properties.group.id' = 'testGroup',
> > > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > > 'format' = 'csv'
> > > > > > )
> > > > > >
> > > > > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > > > > >
> > > > > > For variable management, I have some question:
> > > > > >
> > > > > > 1. Is the parameter used within the team? If all teams use a
> > > common
> > > > > > parameter, it
> > > > > > needs to be defined for each team, right?
> > > > > > 2. Which users can modify variables? Can all users modify all
> > > > > variables of
> > > > > > their team?
> > > > > > After the variable is modified incorrectly, many jobs may be
> > > > > affected. So I
> > > > > > suggest that
> > > > > > only team admin or system admin have permission to modify, add
> > and
> > > > > delete
> > > > > > variables.
> > > > > >
> > > > > > Best
> > > > > > fanrui
> > > > >
> > > > >
> > > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <benjobx@gmail.com>
> > > wrote:
> > > > >
> > > > > > hi 阿穆:
> > > > > >
> > > > > >
> > > > > > Thanks for starting a very useful discussion, I have a some
> > > > questions:
> > > > > >
> > > > > >
> > > > > > 1. The access boundary problem of variable management, is this
> > all
> > > > > > people in the platform can access these variables( no matter
> who
> > > > > > defines) or that only team members can be access these
> > variables.
> > > > > >
> > > > > > 2. There are two usage scenarios for variables that I know of,
> > one
> > > > is
> > > > > > in the entry parameter of main, and the other is in flinksql.
> > Are
> > > > both
> > > > > > supported, or only supported main entry ?
> > > > > >
> > > > > > about first question, my opinion:
> > > > > >
> > > > > > 1. The variable management is associated with the team. After
> > the
> > > > user
> > > > > > logs in to the system, he must select a team (a user can
> belong
> > to
> > > > one
> > > > > > or more teams), The variable belongs to a team
> > > > > >
> > > > > > 2. All resources between teams are isolated, which means that
> > > > > > variables are also isolated. In short, members of a team can
> use
> > > and
> > > > > > define these variables, which are invisible to other members,
> > and
> > > > > > other members cannot use these variables.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Welcome to your views
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Huajie Wang
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com> 于2022年10月7日周五 14:18写道：
> > > > >
> > > > > > > Is it possible to do this variable support in sql as well
> > > > > > >
> > > > > > > 阿穆 <305343894@qq.com.invalid> 于2022年10月7日周五 13:54写道：
> > > > > > >
> > > > > > > > Hi guys,
> > > > > > > >
> > > > > > > >
> > > > > > > > I want to start a discussion about the variable
> > > management
> > > > > module. Please
> > > > > > > > see this issue issue-1477.
> > > > > > > >
> > > > > > > >
> > > > > > > > Background:
> > > > > > > >
> > > > > > > >
> > > > > > > > In the actual production environment, Flink jobs are
> > > > > generally complex,
> > > > > > > > and usually require multiple external components.
> For
> > > > > example, Flink jobs
> > > > > > > > consume data from Kafka, then connect external
> > > components
> > > > > such as HBase
> > > > > > > or
> > > > > > > > Redis to obtain additional business information, and
> > > then
> > > > > write it to the
> > > > > > > > downstream external components. There are the
> > following
> > > > > problems:
> > > > > > > >
> > > > > > > >
> > > > > > > > 1. The connection information of external
> components,
> > > such
> > > > > as IP, port
> > > > > > > and
> > > > > > > > user password, needs to be configured in the
> > application
> > > > > args and
> > > > > > > > transferred to the Flink job, so that the connection
> > > > > information of
> > > > > > > > external components is distributed in multiple
> > > > > applications. Once the
> > > > > > > > connection information of external components
> changes,
> > > > many
> > > > > application
> > > > > > > > args parameters need to be modified, which will lead
> > to
> > > > > high operation
> > > > > > > and
> > > > > > > > maintenance costs.
> > > > > > > > 2. There are many people in the team developing
> > > real-time
> > > > > computing jobs.
> > > > > > > > There is no uniform specification for the connection
> > > > > information of the
> > > > > > > > external components passed to the job, resulting in
> > > > > different parameter
> > > > > > > > names of the same component. This is difficult to
> > count
> > > > > which external
> > > > > > > > components are dependent.
> > > > > > > > 3. In production practice, there are usually
> multiple
> > > sets
> > > > > of
> > > > > > > > environments, such as test environment and
> production
> > > > > environment. It is
> > > > > > > > not intuitive to judge whether a component belongs
> to
> > a
> > > > > test environment
> > > > > > > or
> > > > > > > > a production environment simply through IP and
> ports.
> > > Even
> > > > > if it can be
> > > > > > > > judged, there are some omissions. In this way, the
> > > > > connection information
> > > > > > > > online to the production environment may be external
> > > > > components of the
> > > > > > > test
> > > > > > > > environment, or components of the production
> > environment
> > > > are
> > > > > > > inadvertently
> > > > > > > > configured during testing, leading to production
> > > failures.
> > > > > > > >
> > > > > > > >
> > > > > > > > Proposal:
> > > > > > > >
> > > > > > > >
> > > > > > > > 1. Create a unified module to manage variables.
> Users
> > > can
> > > > > define their
> > > > > > > own
> > > > > > > > variables in this module. Variable information
> > includes
> > > > > variable code
> > > > > > > > (passed to Flink program as parameter name),
> variable
> > > > > value, variable
> > > > > > name,
> > > > > > > > variable description, and which applications depend
> on
> > > > > > > > 2. In the same Team module, the variable code or
> > > variable
> > > > > name is unique
> > > > > > > > - By using placeholders, such as adding -- kafka
> > > {mykafka}
> > > > > to the args of
> > > > > > > > the application, mykafka is the variable code of a
> > > > variable
> > > > > > > > - Search by variable code or variable name. After a
> > > > > variable is selected,
> > > > > > > > the parameter transfer method is displayed
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > If you have any suggestions, welcome to participate
> in
> > > the
> > > > > discussion on
> > > > > > > > the mail list or issue, I look forward to your
> > feedback.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best wishes
> > > > > > >
> > > >
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Chunjin Mu <ma...@gmail.com>.
The variable management module has three properties, including variable
code, variable name, and description, For example, the variable code is
"external.kafka.collect.brokers", and the variable name is "外部采集整合kafka".
When we created the application, I forgot what the variable code was, but I
knew it was called "采集整合", so I can search Enter the "采集整合" keywords in the
box, and of course you can also search for kafka keywords.
At the same time, I want to know more detailed information about this
kafka, such as who the operator is, what is the data retention period, and
how big is the cluster size, but I can't use the description to search, so
I can only search or variable code or variable name, and the variable name
is also for a simple display.
Although external.kafka.collect.brokers already contains a lot of
information, I think this is just a naming convention. When we discuss this
variable, we will discuss it by using the "采集整合 kafka"
We can discuss whether there is a problem with this design, what do you
think?
Rui Fan <19...@gmail.com> 于2022年10月10日周一 11:45写道：
> Hi huajie:
>
> > If it is manually entered, it will always be treated as ordinary text.
>
> Users always copy some code from an old job to a new job.
> I'm afraid it might not meet the user's expectations.
>
> Best,
> fanrui
>
> On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <be...@gmail.com> wrote:
>
> > hi 田向阳:
> >
> > The current discussion is about variable management, which is a
> relatively
> > independent function, No other modules involved, you can check the thread
> > [1]
> >
> >
> > [1] thread:
> > https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
> >
> >
> > Best,
> > Huajie Wang
> >
> >
> >
> > 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> >
> > > is this module include a resource management cente r where jars can be
> > > chosen to provide to create a job
> > >
> > >
> > >
> > > ---- 回复的原邮件 ----
> > > | 发件人 | Huajie Wang<be...@gmail.com> |
> > > | 日期 | 2022年10月08日 12:07 |
> > > | 收件人 | dev@streampark.apache.org<de...@streampark.apache.org> |
> > > | 抄送至 | |
> > > | 主题 | Re: [DISCUSS][issue-1477] Add variable management module |
> > > hi 阿穆:
> > > Thank you for the clarification, My question is answered, but now I
> have
> > > one more concern,
> > > I'm currently using {var} for variables, and I'm worried that one of
> the
> > > user's parameters includes "{" and "}", e.g :
> > >
> > > --password 'mypass{123}abc'
> > >
> > > So how do we parse parse this? How about ${var}
> > >
> > >
> > >
> > > Best,
> > > Huajie Wang
> > >
> > >
> > >
> > > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> > >
> > > > Hi, monster, Huajie Wang, Rui Fan:
> > > >
> > > > Thanks for your reply.
> > > >
> > > > I have integrated your replies. I really don't pay much attention to
> > SQL,
> > > > so I will do the following.
> > > >
> > > > 1.Each team has independent variables that do not affect each other
> and
> > > > are isolated from each other, but the same variable is unique under
> the
> > > > same team, so you need to specify the team when creating variables
> > (users
> > > > may belong to multiple teams, and the teams of the current user will
> be
> > > > displayed here for users to choose)
> > > > 2.For Flink jar jobs, there are two ways to pass variables
> > > > - By using placeholders, such as adding --kafka {mykafka} to the args
> > of
> > > > the application, 'mykafka' is the variable code of a variable
> > > > - Search by variable code or variable name. After a variable is
> > selected,
> > > > the parameter transfer method is displayed
> > > > 3.Flink sql will use variables as placeholders, E.g:
> > > >
> > > > CREATE TABLE KafkaTableA (
> > > > `user_id` BIGINT,
> > > > `item_id` BIGINT,
> > > > `type` STRING
> > > > ) WITH (
> > > > 'connector' = 'kafka',
> > > > 'topic' = 'user_behaviorA',
> > > > 'properties.bootstrap.servers' =
> '{kafka_clusterA}',
> > > --
> > > > 'kafka_ ClusterA' is the variable code
> > > > 'properties.group.id' = 'testGroupA',
> > > > 'scan.startup.mode' = 'earliest-offset',
> > > >
> > > > CREATE TABLE KafkaTableB (
> > > > `user_id` BIGINT,
> > > > `item_id` BIGINT
> > > > ) WITH (
> > > > 'connector' = 'kafka',
> > > > 'topic' = 'user_behaviorB',
> > > > 'properties.bootstrap.servers' =
> '{kafka_clusterB}',
> > > --
> > > > 'kafka_ ClusterB' is the variable code
> > > > 'properties.group.id' = 'testGroupB',
> > > > 'scan.startup.mode' = 'earliest-offset',
> > > >
> > > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM
> > > > KafkaTableB -- 'type' is the variable code
> > > >
> > > > 4.In the future, we can consider setting some system built-in
> variables
> > > >
> > > > Best,
> > > > ChunJin Mu
> > > >
> > > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <1996fanrui@gmail.com>
> > wrote:
> > > >
> > > > > Hi 阿穆:
> > > > >
> > > > > It's a great proposal.
> > > > >
> > > > > For usage scenarios, I strongly recommend support for replacing
> > > > variables
> > > > > in sql. E.g:
> > > > >
> > > > > CREATE TABLE KafkaTable (
> > > > > `user_id` BIGINT,
> > > > > `item_id` BIGINT
> > > > > ) WITH (
> > > > > 'connector' = 'kafka',
> > > > > 'topic' = 'user_behavior',
> > > > > 'properties.bootstrap.servers' =
> > > '{kafka_clusterA}',
> > > > > 'properties.group.id' = 'testGroup',
> > > > > 'scan.startup.mode' = 'earliest-offset',
> > > > > 'format' = 'csv'
> > > > > )
> > > > >
> > > > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > > > >
> > > > > For variable management, I have some question:
> > > > >
> > > > > 1. Is the parameter used within the team? If all teams use a
> > common
> > > > > parameter, it
> > > > > needs to be defined for each team, right?
> > > > > 2. Which users can modify variables? Can all users modify all
> > > > variables of
> > > > > their team?
> > > > > After the variable is modified incorrectly, many jobs may be
> > > > affected. So I
> > > > > suggest that
> > > > > only team admin or system admin have permission to modify, add
> and
> > > > delete
> > > > > variables.
> > > > >
> > > > > Best
> > > > > fanrui
> > > >
> > > >
> > > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <benjobx@gmail.com>
> > wrote:
> > > >
> > > > > hi 阿穆:
> > > > >
> > > > >
> > > > > Thanks for starting a very useful discussion, I have a some
> > > questions:
> > > > >
> > > > >
> > > > > 1. The access boundary problem of variable management, is this
> all
> > > > > people in the platform can access these variables( no matter who
> > > > > defines) or that only team members can be access these
> variables.
> > > > >
> > > > > 2. There are two usage scenarios for variables that I know of,
> one
> > > is
> > > > > in the entry parameter of main, and the other is in flinksql.
> Are
> > > both
> > > > > supported, or only supported main entry ?
> > > > >
> > > > > about first question, my opinion:
> > > > >
> > > > > 1. The variable management is associated with the team. After
> the
> > > user
> > > > > logs in to the system, he must select a team (a user can belong
> to
> > > one
> > > > > or more teams), The variable belongs to a team
> > > > >
> > > > > 2. All resources between teams are isolated, which means that
> > > > > variables are also isolated. In short, members of a team can use
> > and
> > > > > define these variables, which are invisible to other members,
> and
> > > > > other members cannot use these variables.
> > > > >
> > > > >
> > > > >
> > > > > Welcome to your views
> > > > >
> > > > >
> > > > > Best,
> > > > > Huajie Wang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 陈卓宇 <monster.zhuoyuchen@gmail.com> 于2022年10月7日周五 14:18写道：
> > > >
> > > > > > Is it possible to do this variable support in sql as well
> > > > > >
> > > > > > 阿穆 <305343894@qq.com.invalid> 于2022年10月7日周五 13:54写道：
> > > > > >
> > > > > > > Hi guys,
> > > > > > >
> > > > > > >
> > > > > > > I want to start a discussion about the variable
> > management
> > > > module. Please
> > > > > > > see this issue issue-1477.
> > > > > > >
> > > > > > >
> > > > > > > Background:
> > > > > > >
> > > > > > >
> > > > > > > In the actual production environment, Flink jobs are
> > > > generally complex,
> > > > > > > and usually require multiple external components. For
> > > > example, Flink jobs
> > > > > > > consume data from Kafka, then connect external
> > components
> > > > such as HBase
> > > > > > or
> > > > > > > Redis to obtain additional business information, and
> > then
> > > > write it to the
> > > > > > > downstream external components. There are the
> following
> > > > problems:
> > > > > > >
> > > > > > >
> > > > > > > 1. The connection information of external components,
> > such
> > > > as IP, port
> > > > > > and
> > > > > > > user password, needs to be configured in the
> application
> > > > args and
> > > > > > > transferred to the Flink job, so that the connection
> > > > information of
> > > > > > > external components is distributed in multiple
> > > > applications. Once the
> > > > > > > connection information of external components changes,
> > > many
> > > > application
> > > > > > > args parameters need to be modified, which will lead
> to
> > > > high operation
> > > > > > and
> > > > > > > maintenance costs.
> > > > > > > 2. There are many people in the team developing
> > real-time
> > > > computing jobs.
> > > > > > > There is no uniform specification for the connection
> > > > information of the
> > > > > > > external components passed to the job, resulting in
> > > > different parameter
> > > > > > > names of the same component. This is difficult to
> count
> > > > which external
> > > > > > > components are dependent.
> > > > > > > 3. In production practice, there are usually multiple
> > sets
> > > > of
> > > > > > > environments, such as test environment and production
> > > > environment. It is
> > > > > > > not intuitive to judge whether a component belongs to
> a
> > > > test environment
> > > > > > or
> > > > > > > a production environment simply through IP and ports.
> > Even
> > > > if it can be
> > > > > > > judged, there are some omissions. In this way, the
> > > > connection information
> > > > > > > online to the production environment may be external
> > > > components of the
> > > > > > test
> > > > > > > environment, or components of the production
> environment
> > > are
> > > > > > inadvertently
> > > > > > > configured during testing, leading to production
> > failures.
> > > > > > >
> > > > > > >
> > > > > > > Proposal:
> > > > > > >
> > > > > > >
> > > > > > > 1. Create a unified module to manage variables. Users
> > can
> > > > define their
> > > > > > own
> > > > > > > variables in this module. Variable information
> includes
> > > > variable code
> > > > > > > (passed to Flink program as parameter name), variable
> > > > value, variable
> > > > > name,
> > > > > > > variable description, and which applications depend on
> > > > > > > 2. In the same Team module, the variable code or
> > variable
> > > > name is unique
> > > > > > > - By using placeholders, such as adding -- kafka
> > {mykafka}
> > > > to the args of
> > > > > > > the application, mykafka is the variable code of a
> > > variable
> > > > > > > - Search by variable code or variable name. After a
> > > > variable is selected,
> > > > > > > the parameter transfer method is displayed
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > If you have any suggestions, welcome to participate in
> > the
> > > > discussion on
> > > > > > > the mail list or issue, I look forward to your
> feedback.
> > > > > > >
> > > > > > >
> > > > > > > Best wishes
> > > > > >
> > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Rui Fan <19...@gmail.com>.
Hi huajie:
> If it is manually entered, it will always be treated as ordinary text.
Users always copy some code from an old job to a new job.
I'm afraid it might not meet the user's expectations.
Best,
fanrui
On Sat, Oct 8, 2022 at 3:12 PM Huajie Wang <be...@gmail.com> wrote:
> hi 田向阳:
>
> The current discussion is about variable management, which is a relatively
> independent function, No other modules involved, you can check the thread
> [1]
>
>
> [1] thread:
> https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
>
>
> Best,
> Huajie Wang
>
>
>
> 田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
>
> > is this module include a resource management cente r where jars can be
> > chosen to provide to create a job
> >
> >
> >
> > ---- 回复的原邮件 ----
> > | 发件人 | Huajie Wang<be...@gmail.com> |
> > | 日期 | 2022年10月08日 12:07 |
> > | 收件人 | dev@streampark.apache.org<de...@streampark.apache.org> |
> > | 抄送至 | |
> > | 主题 | Re: [DISCUSS][issue-1477] Add variable management module |
> > hi 阿穆:
> > Thank you for the clarification, My question is answered, but now I have
> > one more concern,
> > I'm currently using {var} for variables, and I'm worried that one of the
> > user's parameters includes "{" and "}", e.g :
> >
> > --password 'mypass{123}abc'
> >
> > So how do we parse parse this? How about ${var}
> >
> >
> >
> > Best,
> > Huajie Wang
> >
> >
> >
> > 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
> >
> > > Hi, monster, Huajie Wang, Rui Fan:
> > >
> > > Thanks for your reply.
> > >
> > > I have integrated your replies. I really don't pay much attention to
> SQL,
> > > so I will do the following.
> > >
> > > 1.Each team has independent variables that do not affect each other and
> > > are isolated from each other, but the same variable is unique under the
> > > same team, so you need to specify the team when creating variables
> (users
> > > may belong to multiple teams, and the teams of the current user will be
> > > displayed here for users to choose)
> > > 2.For Flink jar jobs, there are two ways to pass variables
> > > - By using placeholders, such as adding --kafka {mykafka} to the args
> of
> > > the application, 'mykafka' is the variable code of a variable
> > > - Search by variable code or variable name. After a variable is
> selected,
> > > the parameter transfer method is displayed
> > > 3.Flink sql will use variables as placeholders, E.g:
> > >
> > > CREATE TABLE KafkaTableA (
> > > `user_id` BIGINT,
> > > `item_id` BIGINT,
> > > `type` STRING
> > > ) WITH (
> > > 'connector' = 'kafka',
> > > 'topic' = 'user_behaviorA',
> > > 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > --
> > > 'kafka_ ClusterA' is the variable code
> > > 'properties.group.id' = 'testGroupA',
> > > 'scan.startup.mode' = 'earliest-offset',
> > >
> > > CREATE TABLE KafkaTableB (
> > > `user_id` BIGINT,
> > > `item_id` BIGINT
> > > ) WITH (
> > > 'connector' = 'kafka',
> > > 'topic' = 'user_behaviorB',
> > > 'properties.bootstrap.servers' = '{kafka_clusterB}',
> > --
> > > 'kafka_ ClusterB' is the variable code
> > > 'properties.group.id' = 'testGroupB',
> > > 'scan.startup.mode' = 'earliest-offset',
> > >
> > > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM
> > > KafkaTableB -- 'type' is the variable code
> > >
> > > 4.In the future, we can consider setting some system built-in variables
> > >
> > > Best,
> > > ChunJin Mu
> > >
> > > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <1996fanrui@gmail.com>
> wrote:
> > >
> > > > Hi 阿穆:
> > > >
> > > > It's a great proposal.
> > > >
> > > > For usage scenarios, I strongly recommend support for replacing
> > > variables
> > > > in sql. E.g:
> > > >
> > > > CREATE TABLE KafkaTable (
> > > > `user_id` BIGINT,
> > > > `item_id` BIGINT
> > > > ) WITH (
> > > > 'connector' = 'kafka',
> > > > 'topic' = 'user_behavior',
> > > > 'properties.bootstrap.servers' =
> > '{kafka_clusterA}',
> > > > 'properties.group.id' = 'testGroup',
> > > > 'scan.startup.mode' = 'earliest-offset',
> > > > 'format' = 'csv'
> > > > )
> > > >
> > > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > > >
> > > > For variable management, I have some question:
> > > >
> > > > 1. Is the parameter used within the team? If all teams use a
> common
> > > > parameter, it
> > > > needs to be defined for each team, right?
> > > > 2. Which users can modify variables? Can all users modify all
> > > variables of
> > > > their team?
> > > > After the variable is modified incorrectly, many jobs may be
> > > affected. So I
> > > > suggest that
> > > > only team admin or system admin have permission to modify, add and
> > > delete
> > > > variables.
> > > >
> > > > Best
> > > > fanrui
> > >
> > >
> > > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <benjobx@gmail.com>
> wrote:
> > >
> > > > hi 阿穆:
> > > >
> > > >
> > > > Thanks for starting a very useful discussion, I have a some
> > questions:
> > > >
> > > >
> > > > 1. The access boundary problem of variable management, is this all
> > > > people in the platform can access these variables( no matter who
> > > > defines) or that only team members can be access these variables.
> > > >
> > > > 2. There are two usage scenarios for variables that I know of, one
> > is
> > > > in the entry parameter of main, and the other is in flinksql. Are
> > both
> > > > supported, or only supported main entry ?
> > > >
> > > > about first question, my opinion:
> > > >
> > > > 1. The variable management is associated with the team. After the
> > user
> > > > logs in to the system, he must select a team (a user can belong to
> > one
> > > > or more teams), The variable belongs to a team
> > > >
> > > > 2. All resources between teams are isolated, which means that
> > > > variables are also isolated. In short, members of a team can use
> and
> > > > define these variables, which are invisible to other members, and
> > > > other members cannot use these variables.
> > > >
> > > >
> > > >
> > > > Welcome to your views
> > > >
> > > >
> > > > Best,
> > > > Huajie Wang
> > > >
> > > >
> > > >
> > > >
> > > > 陈卓宇 <monster.zhuoyuchen@gmail.com> 于2022年10月7日周五 14:18写道：
> > >
> > > > > Is it possible to do this variable support in sql as well
> > > > >
> > > > > 阿穆 <305343894@qq.com.invalid> 于2022年10月7日周五 13:54写道：
> > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > >
> > > > > > I want to start a discussion about the variable
> management
> > > module. Please
> > > > > > see this issue issue-1477.
> > > > > >
> > > > > >
> > > > > > Background:
> > > > > >
> > > > > >
> > > > > > In the actual production environment, Flink jobs are
> > > generally complex,
> > > > > > and usually require multiple external components. For
> > > example, Flink jobs
> > > > > > consume data from Kafka, then connect external
> components
> > > such as HBase
> > > > > or
> > > > > > Redis to obtain additional business information, and
> then
> > > write it to the
> > > > > > downstream external components. There are the following
> > > problems:
> > > > > >
> > > > > >
> > > > > > 1. The connection information of external components,
> such
> > > as IP, port
> > > > > and
> > > > > > user password, needs to be configured in the application
> > > args and
> > > > > > transferred to the Flink job, so that the connection
> > > information of
> > > > > > external components is distributed in multiple
> > > applications. Once the
> > > > > > connection information of external components changes,
> > many
> > > application
> > > > > > args parameters need to be modified, which will lead to
> > > high operation
> > > > > and
> > > > > > maintenance costs.
> > > > > > 2. There are many people in the team developing
> real-time
> > > computing jobs.
> > > > > > There is no uniform specification for the connection
> > > information of the
> > > > > > external components passed to the job, resulting in
> > > different parameter
> > > > > > names of the same component. This is difficult to count
> > > which external
> > > > > > components are dependent.
> > > > > > 3. In production practice, there are usually multiple
> sets
> > > of
> > > > > > environments, such as test environment and production
> > > environment. It is
> > > > > > not intuitive to judge whether a component belongs to a
> > > test environment
> > > > > or
> > > > > > a production environment simply through IP and ports.
> Even
> > > if it can be
> > > > > > judged, there are some omissions. In this way, the
> > > connection information
> > > > > > online to the production environment may be external
> > > components of the
> > > > > test
> > > > > > environment, or components of the production environment
> > are
> > > > > inadvertently
> > > > > > configured during testing, leading to production
> failures.
> > > > > >
> > > > > >
> > > > > > Proposal:
> > > > > >
> > > > > >
> > > > > > 1. Create a unified module to manage variables. Users
> can
> > > define their
> > > > > own
> > > > > > variables in this module. Variable information includes
> > > variable code
> > > > > > (passed to Flink program as parameter name), variable
> > > value, variable
> > > > name,
> > > > > > variable description, and which applications depend on
> > > > > > 2. In the same Team module, the variable code or
> variable
> > > name is unique
> > > > > > - By using placeholders, such as adding -- kafka
> {mykafka}
> > > to the args of
> > > > > > the application, mykafka is the variable code of a
> > variable
> > > > > > - Search by variable code or variable name. After a
> > > variable is selected,
> > > > > > the parameter transfer method is displayed
> > > > > >
> > > > > >
> > > > > >
> > > > > > If you have any suggestions, welcome to participate in
> the
> > > discussion on
> > > > > > the mail list or issue, I look forward to your feedback.
> > > > > >
> > > > > >
> > > > > > Best wishes
> > > > >
> >
>
Re: [DISCUSS][issue-1477] Add variable management module
Posted by Huajie Wang <be...@gmail.com>.
hi 田向阳:
The current discussion is about variable management, which is a relatively
independent function, No other modules involved, you can check the thread
[1]
[1] thread: https://lists.apache.org/thread/s9sjpkjn6spvq07bk34cfd3vln7rjo0m
Best,
Huajie Wang
田向阳 <lu...@163.com> 于2022年10月8日周六 13:13写道：
> is this module include a resource management cente r where jars can be
> chosen to provide to create a job
>
>
>
> ---- 回复的原邮件 ----
> | 发件人 | Huajie Wang<be...@gmail.com> |
> | 日期 | 2022年10月08日 12:07 |
> | 收件人 | dev@streampark.apache.org<de...@streampark.apache.org> |
> | 抄送至 | |
> | 主题 | Re: [DISCUSS][issue-1477] Add variable management module |
> hi 阿穆:
> Thank you for the clarification, My question is answered, but now I have
> one more concern,
> I'm currently using {var} for variables, and I'm worried that one of the
> user's parameters includes "{" and "}", e.g :
>
> --password 'mypass{123}abc'
>
> So how do we parse parse this? How about ${var}
>
>
>
> Best,
> Huajie Wang
>
>
>
> 阿穆 <30...@qq.com.invalid> 于2022年10月8日周六 11:36写道：
>
> > Hi, monster, Huajie Wang, Rui Fan:
> >
> > Thanks for your reply.
> >
> > I have integrated your replies. I really don't pay much attention to SQL,
> > so I will do the following.
> >
> > 1.Each team has independent variables that do not affect each other and
> > are isolated from each other, but the same variable is unique under the
> > same team, so you need to specify the team when creating variables (users
> > may belong to multiple teams, and the teams of the current user will be
> > displayed here for users to choose)
> > 2.For Flink jar jobs, there are two ways to pass variables
> > - By using placeholders, such as adding --kafka {mykafka} to the args of
> > the application, 'mykafka' is the variable code of a variable
> > - Search by variable code or variable name. After a variable is selected,
> > the parameter transfer method is displayed
> > 3.Flink sql will use variables as placeholders, E.g:
> >
> > CREATE TABLE KafkaTableA (
> > `user_id` BIGINT,
> > `item_id` BIGINT,
> > `type` STRING
> > ) WITH (
> > 'connector' = 'kafka',
> > 'topic' = 'user_behaviorA',
> > 'properties.bootstrap.servers' = '{kafka_clusterA}',
> --
> > 'kafka_ ClusterA' is the variable code
> > 'properties.group.id' = 'testGroupA',
> > 'scan.startup.mode' = 'earliest-offset',
> >
> > CREATE TABLE KafkaTableB (
> > `user_id` BIGINT,
> > `item_id` BIGINT
> > ) WITH (
> > 'connector' = 'kafka',
> > 'topic' = 'user_behaviorB',
> > 'properties.bootstrap.servers' = '{kafka_clusterB}',
> --
> > 'kafka_ ClusterB' is the variable code
> > 'properties.group.id' = 'testGroupB',
> > 'scan.startup.mode' = 'earliest-offset',
> >
> > INSERT INTO KafkaTableA SELECT user_id, item_id, '{type}' FROM
> > KafkaTableB -- 'type' is the variable code
> >
> > 4.In the future, we can consider setting some system built-in variables
> >
> > Best,
> > ChunJin Mu
> >
> > On Fri, Oct 7, 2022 at 8:27 PM Rui Fan <1996fanrui@gmail.com> wrote:
> >
> > > Hi 阿穆:
> > >
> > > It's a great proposal.
> > >
> > > For usage scenarios, I strongly recommend support for replacing
> > variables
> > > in sql. E.g:
> > >
> > > CREATE TABLE KafkaTable (
> > > `user_id` BIGINT,
> > > `item_id` BIGINT
> > > ) WITH (
> > > 'connector' = 'kafka',
> > > 'topic' = 'user_behavior',
> > > 'properties.bootstrap.servers' =
> '{kafka_clusterA}',
> > > 'properties.group.id' = 'testGroup',
> > > 'scan.startup.mode' = 'earliest-offset',
> > > 'format' = 'csv'
> > > )
> > >
> > > Note: 'properties.bootstrap.servers' = '{kafka_clusterA}',
> > >
> > > For variable management, I have some question:
> > >
> > > 1. Is the parameter used within the team? If all teams use a common
> > > parameter, it
> > > needs to be defined for each team, right?
> > > 2. Which users can modify variables? Can all users modify all
> > variables of
> > > their team?
> > > After the variable is modified incorrectly, many jobs may be
> > affected. So I
> > > suggest that
> > > only team admin or system admin have permission to modify, add and
> > delete
> > > variables.
> > >
> > > Best
> > > fanrui
> >
> >
> > On Fri, Oct 7, 2022 at 7:01 PM Huajie Wang <benjobx@gmail.com> wrote:
> >
> > > hi 阿穆:
> > >
> > >
> > > Thanks for starting a very useful discussion, I have a some
> questions:
> > >
> > >
> > > 1. The access boundary problem of variable management, is this all
> > > people in the platform can access these variables( no matter who
> > > defines) or that only team members can be access these variables.
> > >
> > > 2. There are two usage scenarios for variables that I know of, one
> is
> > > in the entry parameter of main, and the other is in flinksql. Are
> both
> > > supported, or only supported main entry ?
> > >
> > > about first question, my opinion:
> > >
> > > 1. The variable management is associated with the team. After the
> user
> > > logs in to the system, he must select a team (a user can belong to
> one
> > > or more teams), The variable belongs to a team
> > >
> > > 2. All resources between teams are isolated, which means that
> > > variables are also isolated. In short, members of a team can use and
> > > define these variables, which are invisible to other members, and
> > > other members cannot use these variables.
> > >
> > >
> > >
> > > Welcome to your views
> > >
> > >
> > > Best,
> > > Huajie Wang
> > >
> > >
> > >
> > >
> > > 陈卓宇 <monster.zhuoyuchen@gmail.com> 于2022年10月7日周五 14:18写道：
> >
> > > > Is it possible to do this variable support in sql as well
> > > >
> > > > 阿穆 <305343894@qq.com.invalid> 于2022年10月7日周五 13:54写道：
> > > >
> > > > > Hi guys,
> > > > >
> > > > >
> > > > > I want to start a discussion about the variable management
> > module. Please
> > > > > see this issue issue-1477.
> > > > >
> > > > >
> > > > > Background:
> > > > >
> > > > >
> > > > > In the actual production environment, Flink jobs are
> > generally complex,
> > > > > and usually require multiple external components. For
> > example, Flink jobs
> > > > > consume data from Kafka, then connect external components
> > such as HBase
> > > > or
> > > > > Redis to obtain additional business information, and then
> > write it to the
> > > > > downstream external components. There are the following
> > problems:
> > > > >
> > > > >
> > > > > 1. The connection information of external components, such
> > as IP, port
> > > > and
> > > > > user password, needs to be configured in the application
> > args and
> > > > > transferred to the Flink job, so that the connection
> > information of
> > > > > external components is distributed in multiple
> > applications. Once the
> > > > > connection information of external components changes,
> many
> > application
> > > > > args parameters need to be modified, which will lead to
> > high operation
> > > > and
> > > > > maintenance costs.
> > > > > 2. There are many people in the team developing real-time
> > computing jobs.
> > > > > There is no uniform specification for the connection
> > information of the
> > > > > external components passed to the job, resulting in
> > different parameter
> > > > > names of the same component. This is difficult to count
> > which external
> > > > > components are dependent.
> > > > > 3. In production practice, there are usually multiple sets
> > of
> > > > > environments, such as test environment and production
> > environment. It is
> > > > > not intuitive to judge whether a component belongs to a
> > test environment
> > > > or
> > > > > a production environment simply through IP and ports. Even
> > if it can be
> > > > > judged, there are some omissions. In this way, the
> > connection information
> > > > > online to the production environment may be external
> > components of the
> > > > test
> > > > > environment, or components of the production environment
> are
> > > > inadvertently
> > > > > configured during testing, leading to production failures.
> > > > >
> > > > >
> > > > > Proposal:
> > > > >
> > > > >
> > > > > 1. Create a unified module to manage variables. Users can
> > define their
> > > > own
> > > > > variables in this module. Variable information includes
> > variable code
> > > > > (passed to Flink program as parameter name), variable
> > value, variable
> > > name,
> > > > > variable description, and which applications depend on
> > > > > 2. In the same Team module, the variable code or variable
> > name is unique
> > > > > - By using placeholders, such as adding -- kafka {mykafka}
> > to the args of
> > > > > the application, mykafka is the variable code of a
> variable
> > > > > - Search by variable code or variable name. After a
> > variable is selected,
> > > > > the parameter transfer method is displayed
> > > > >
> > > > >
> > > > >
> > > > > If you have any suggestions, welcome to participate in the
> > discussion on
> > > > > the mail list or issue, I look forward to your feedback.
> > > > >
> > > > >
> > > > > Best wishes
> > > >
>