You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user-zh@flink.apache.org by gangzi <11...@qq.com.INVALID> on 2022/04/11 11:55:36 UTC
flink命令行参数不生效问题
我用命令提交作业：flink run -t yarn-per-job -ynm SocketWordCount -yqu root.root -d -n SocketWindowWordCount.jar --hostname 10.199.0.97 --port 9878。结果作业提交成功之后发现 -ynm和-yqu不生效。后来通过查看源码发现是因为如果指定了 -t，那么-y开头的所有参数都不生效了，因为-y系列参数是在FlinkYarnSessionCli中解析的，而源码中：public CustomCommandLine validateAndGetActiveCommandLine(CommandLine commandLine) {
LOG.debug("Custom commandlines: {}", customCommandLines);
for (CustomCommandLine cli : customCommandLines) {
LOG.debug(
"Checking custom commandline {}, isActive: {}", cli, cli.isActive(commandLine));
if (cli.isActive(commandLine)) {
return cli;
}
}
throw new IllegalStateException("No valid command-line found.");
}
这段代码返回的是GenericCLI。导致后面的：
final Configuration effectiveConfiguration =
getEffectiveConfiguration(activeCommandLine, commandLine, programOptions, jobJars);
这行代码返回的命令行参数配置只包含了GenericCli中定义的参数。想请教一下，-t和-m设置参数时有啥区别？如何解决上述参数不生效的问题？这是一个bug么？
Re: flink命令行参数不生效问题
Posted by Zhanghao Chen <zh...@outlook.com>.
我知道的比较相关的 FLIP 和 JIRA 有下面两个：
1. https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission
FLIP-73: Introducing Executors for job submission - Apache Flink - The Apache Software Foundation<https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission>
Status. Current state: Accepted. Discussion thread: https://lists.apache.org/thread.html/dc3a541709f96906b43df4155373af1cd09e08c3f105b0bd0ba3fca2@%3Cdev.flink.apache ...
cwiki.apache.org
2. https://issues.apache.org/jira/browse/FLINK-15179 特别是这个 JIRA 的评论区可以看一下
Best,
Zhanghao Chen
________________________________
From: QQ <11...@qq.com.INVALID>
Sent: Tuesday, April 12, 2022 9:19
To: user-zh@flink.apache.org <us...@flink.apache.org>
Subject: Re: flink命令行参数不生效问题
非常感谢解答。关于这套新的统一的cli命令的说明或者FLIP在哪里？
> 2022年4月11日 下午11:49，Zhanghao Chen <zh...@outlook.com> 写道：
>
> 你好，-m 配合 -yxx 的参数是早期 Flink on YARN 的 cli 参数用法，后来社区开始推进一套新的统一的 cli 命令，使用 -t 指定部署形式，并将原先的 cli options 统一动态参数化，比如原先的 -yxx 命令都能从 https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn 找到替代的动态参数。
> Configuration | Apache Flink<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn>
> Key Default Type Description; restart-strategy.fixed-delay.attempts: 1: Integer: The number of times that Flink retries the execution before the job is declared as failed if restart-strategy has been set to fixed-delay.: restart-strategy.fixed-delay.delay
> nightlies.apache.org
>
>
> Best,
> Zhanghao Chen
> ________________________________
> From: gangzi <11...@qq.com.INVALID>
> Sent: Monday, April 11, 2022 19:55
> To: user-zh <us...@flink.apache.org>
> Subject: flink命令行参数不生效问题
>
> 我用命令提交作业：flink run -t yarn-per-job -ynm SocketWordCount -yqu root.root -d -n SocketWindowWordCount.jar --hostname 10.199.0.97 --port 9878。结果作业提交成功之后发现 -ynm和-yqu不生效。后来通过查看源码发现是因为如果指定了 -t，那么-y开头的所有参数都不生效了，因为-y系列参数是在FlinkYarnSessionCli中解析的，而源码中：public CustomCommandLine validateAndGetActiveCommandLine(CommandLine commandLine) {
> LOG.debug("Custom commandlines: {}", customCommandLines);
> for (CustomCommandLine cli : customCommandLines) {
> LOG.debug(
> "Checking custom commandline {}, isActive: {}", cli, cli.isActive(commandLine));
> if (cli.isActive(commandLine)) {
> return cli;
> }
> }
> throw new IllegalStateException("No valid command-line found.");
> }
> 这段代码返回的是GenericCLI。导致后面的：
> final Configuration effectiveConfiguration =
> getEffectiveConfiguration(activeCommandLine, commandLine, programOptions, jobJars);
> 这行代码返回的命令行参数配置只包含了GenericCli中定义的参数。想请教一下，-t和-m设置参数时有啥区别？如何解决上述参数不生效的问题？这是一个bug么？
Re: flink命令行参数不生效问题
Posted by QQ <11...@qq.com.INVALID>.
非常感谢解答。关于这套新的统一的cli命令的说明或者FLIP在哪里？
> 2022年4月11日 下午11:49，Zhanghao Chen <zh...@outlook.com> 写道：
>
> 你好，-m 配合 -yxx 的参数是早期 Flink on YARN 的 cli 参数用法，后来社区开始推进一套新的统一的 cli 命令，使用 -t 指定部署形式，并将原先的 cli options 统一动态参数化，比如原先的 -yxx 命令都能从 https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn 找到替代的动态参数。
> Configuration | Apache Flink<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn>
> Key Default Type Description; restart-strategy.fixed-delay.attempts: 1: Integer: The number of times that Flink retries the execution before the job is declared as failed if restart-strategy has been set to fixed-delay.: restart-strategy.fixed-delay.delay
> nightlies.apache.org
>
>
> Best,
> Zhanghao Chen
> ________________________________
> From: gangzi <11...@qq.com.INVALID>
> Sent: Monday, April 11, 2022 19:55
> To: user-zh <us...@flink.apache.org>
> Subject: flink命令行参数不生效问题
>
> 我用命令提交作业：flink run -t yarn-per-job -ynm SocketWordCount -yqu root.root -d -n SocketWindowWordCount.jar --hostname 10.199.0.97 --port 9878。结果作业提交成功之后发现 -ynm和-yqu不生效。后来通过查看源码发现是因为如果指定了 -t，那么-y开头的所有参数都不生效了，因为-y系列参数是在FlinkYarnSessionCli中解析的，而源码中：public CustomCommandLine validateAndGetActiveCommandLine(CommandLine commandLine) {
> LOG.debug("Custom commandlines: {}", customCommandLines);
> for (CustomCommandLine cli : customCommandLines) {
> LOG.debug(
> "Checking custom commandline {}, isActive: {}", cli, cli.isActive(commandLine));
> if (cli.isActive(commandLine)) {
> return cli;
> }
> }
> throw new IllegalStateException("No valid command-line found.");
> }
> 这段代码返回的是GenericCLI。导致后面的：
> final Configuration effectiveConfiguration =
> getEffectiveConfiguration(activeCommandLine, commandLine, programOptions, jobJars);
> 这行代码返回的命令行参数配置只包含了GenericCli中定义的参数。想请教一下，-t和-m设置参数时有啥区别？如何解决上述参数不生效的问题？这是一个bug么？
Re: flink命令行参数不生效问题
Posted by Zhanghao Chen <zh...@outlook.com>.
你好，-m 配合 -yxx 的参数是早期 Flink on YARN 的 cli 参数用法，后来社区开始推进一套新的统一的 cli 命令，使用 -t 指定部署形式，并将原先的 cli options 统一动态参数化，比如原先的 -yxx 命令都能从 https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn 找到替代的动态参数。
Configuration | Apache Flink<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn>
Key Default Type Description; restart-strategy.fixed-delay.attempts: 1: Integer: The number of times that Flink retries the execution before the job is declared as failed if restart-strategy has been set to fixed-delay.: restart-strategy.fixed-delay.delay
nightlies.apache.org
Best,
Zhanghao Chen
________________________________
From: gangzi <11...@qq.com.INVALID>
Sent: Monday, April 11, 2022 19:55
To: user-zh <us...@flink.apache.org>
Subject: flink命令行参数不生效问题
我用命令提交作业：flink run -t yarn-per-job -ynm SocketWordCount -yqu root.root -d -n SocketWindowWordCount.jar --hostname 10.199.0.97 --port 9878。结果作业提交成功之后发现 -ynm和-yqu不生效。后来通过查看源码发现是因为如果指定了 -t，那么-y开头的所有参数都不生效了，因为-y系列参数是在FlinkYarnSessionCli中解析的，而源码中：public CustomCommandLine validateAndGetActiveCommandLine(CommandLine commandLine) {
LOG.debug("Custom commandlines: {}", customCommandLines);
for (CustomCommandLine cli : customCommandLines) {
LOG.debug(
"Checking custom commandline {}, isActive: {}", cli, cli.isActive(commandLine));
if (cli.isActive(commandLine)) {
return cli;
}
}
throw new IllegalStateException("No valid command-line found.");
}
这段代码返回的是GenericCLI。导致后面的：
final Configuration effectiveConfiguration =
getEffectiveConfiguration(activeCommandLine, commandLine, programOptions, jobJars);
这行代码返回的命令行参数配置只包含了GenericCli中定义的参数。想请教一下，-t和-m设置参数时有啥区别？如何解决上述参数不生效的问题？这是一个bug么？