You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streampark.apache.org by "StoneForests (via GitHub)" <gi...@apache.org> on 2024/06/07 05:21:52 UTC
[D] flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs [incubator-streampark]
GitHub user StoneForests created a discussion: flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs
我参照streampark官网教程 [原生flink作业参数](https://streampark.apache.org/zh-CN/docs/user-guide/platformBasicUsage#%E5%8E%9F%E7%94%9Fflink%E4%BD%9C%E4%B8%9A%E5%8F%82%E6%95%B0) ,增加了两个动态参数

任务运行起来后,第一个自定义参数生效了,但第二个参数(yarn.provided.lib.dirs)没有生效

我看了flink的日志,其中yarn.provided.lib.dirs出现了两次,均为streampark的默认值,而非streampark界面上自定义值:


想请教一下:在yarn-application模式下,是不是无法自定义yarn.provided.lib.dirs的值?
`hdfs:///streampark/flink/flink-1.14.4/lib;hdfs:///streampark/jars;hdfs:///streampark/plugins;hdfs:///streampark/workspace/<appid>/lib;hdfs://zq-dev-cluster/streampark/shims/flink-1.14 `
如果真是如此,想问一下是否可以放开这个限制,允许yarn-application模式能在任务中指定加载其它目录中的依赖呢,具体场景如下:
1、我们开发了一个flink交换组件jar,该jar依赖了非常多第三方jar,全部整下来合计好几百MByte,我们把这些第三方jar都提前预置到hdfs:///streampark/extlib;中。
2、在使用步骤1的jar建立custom code任务时,允许用户指定yarn.provided.lib.dirs为hdfs:///streampark/extlib;
3、在使用其它的jar,或者flinksql任务时,使用streampark默认的yarn.provided.lib.dirs目录,从而避免去
之所以我们没有把jar全部放到hdfs:///streampark/jars目录中,是为了避免在步骤3中的fiinksql或其它jar的yarn container也去下载不必要的依赖,浪费网络和存储。
期待您的回复,谢谢!
GitHub link: https://github.com/apache/incubator-streampark/discussions/3743
----
This is an automatically sent email for dev@streampark.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampark.apache.org
Re: [D] flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs [incubator-streampark]
Posted by "StoneForests (via GitHub)" <gi...@apache.org>.
GitHub user StoneForests added a comment to the discussion: flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs
> "/streampark/jars" stores the user's yarn-app mode public jar
Thank you for your reply! I am developing a specific task app.jar, and its dependency jar files are all placed in the hdfs://myjar/ directory. When launching app.jar in yarn-application mode, I need to specify that the dependencies are read from the hdfs://myjar/ directory (i.e., setting yarn.provided.lib.dirs=hdfs://myjar/ instead of the default hdfs:///streampark/flink/flink-1.14.4/lib;hdfs:///streampark/jars;hdfs:///streampark/plugins;hdfs:///streampark/workspace/<appid>/lib;hdfs://zq-dev-cluster/streampark/shims/flink-1.14). However, it seems that in StreamPark, yarn.provided.lib.dirs cannot be customized in yarn-application mode. Is there any solution for this?
GitHub link: https://github.com/apache/incubator-streampark/discussions/3743#discussioncomment-10248895
----
This is an automatically sent email for dev@streampark.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampark.apache.org
Re: [D] flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs [incubator-streampark]
Posted by "StoneForests (via GitHub)" <gi...@apache.org>.
GitHub user StoneForests added a comment to the discussion: flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs
> "/streampark/jars" stores the user's yarn-app mode public jar
The example in [Setting Job Parameters](https://streampark.apache.org/docs/get-started/how-to-use/#how-to-use-in-apache-streampark-1) shows user can be customized in yarn-application mode, but it does not work in fact...Is this a bug in StreamPark 2.1.4? Hope for your reply. THANK YOU!
GitHub link: https://github.com/apache/incubator-streampark/discussions/3743#discussioncomment-10248919
----
This is an automatically sent email for dev@streampark.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampark.apache.org
Re: [D] flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs [incubator-streampark]
Posted by "StoneForests (via GitHub)" <gi...@apache.org>.
GitHub user StoneForests edited a discussion: flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs
Java Version
java1.8
Scala Version
2.12
StreamPark Version
2.1.4
Flink Version
1.14.4
deploy mode
yarn-application
我参照streampark官网教程 [原生flink作业参数](https://streampark.apache.org/docs/get-started/how-to-use/#setting-job-parameters) ,增加了两个动态参数

任务运行起来后,第一个自定义参数生效了,但第二个参数(yarn.provided.lib.dirs)没有生效

我看了flink的日志,其中yarn.provided.lib.dirs出现了两次,均为streampark的默认值,而非streampark界面上自定义值:


想请教一下:在yarn-application模式下,是不是无法自定义yarn.provided.lib.dirs的值?
`hdfs:///streampark/flink/flink-1.14.4/lib;hdfs:///streampark/jars;hdfs:///streampark/plugins;hdfs:///streampark/workspace/<appid>/lib;hdfs://zq-dev-cluster/streampark/shims/flink-1.14 `
如果真是如此,想问一下是否可以放开这个限制,允许yarn-application模式能在任务中指定加载其它目录中的依赖呢,具体场景如下:
1、我们开发了一个flink交换组件jar,该jar依赖了非常多第三方jar,如果合成一个shadejar得好几百MByte,所以我们把主jar和依赖jar拆开,把依赖jar预置到hdfs:///streampark/extlib;中。
2、在使用步骤1的主jar建立custom code任务时,允许用户指定yarn.provided.lib.dirs为hdfs:///streampark/extlib;
3、在使用其它的jar建立custom code任务r,或者flink sql任务时,不需要用户指定yarn.provided.lib.dirs,此时streampark会使用默认的yarn.provided.lib.dirs目录。
之所以我们没有把jar全部放到hdfs:///streampark/jars目录中,是为了避免在步骤3中的flink任务也去下载不必要的依赖,浪费网络和存储。
期待您的回复,谢谢!
GitHub link: https://github.com/apache/incubator-streampark/discussions/3743
----
This is an automatically sent email for dev@streampark.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampark.apache.org
Re: [D] flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs [incubator-streampark]
Posted by "StoneForests (via GitHub)" <gi...@apache.org>.
GitHub user StoneForests edited a discussion: flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs
Java Version
java1.8
Scala Version
2.12
StreamPark Version
2.1.4
Flink Version
1.14.4
deploy mode
yarn-application
我参照streampark官网教程 [原生flink作业参数](https://streampark.apache.org/zh-CN/docs/user-guide/platformBasicUsage#%E5%8E%9F%E7%94%9Fflink%E4%BD%9C%E4%B8%9A%E5%8F%82%E6%95%B0) ,增加了两个动态参数

任务运行起来后,第一个自定义参数生效了,但第二个参数(yarn.provided.lib.dirs)没有生效

我看了flink的日志,其中yarn.provided.lib.dirs出现了两次,均为streampark的默认值,而非streampark界面上自定义值:


想请教一下:在yarn-application模式下,是不是无法自定义yarn.provided.lib.dirs的值?
`hdfs:///streampark/flink/flink-1.14.4/lib;hdfs:///streampark/jars;hdfs:///streampark/plugins;hdfs:///streampark/workspace/<appid>/lib;hdfs://zq-dev-cluster/streampark/shims/flink-1.14 `
如果真是如此,想问一下是否可以放开这个限制,允许yarn-application模式能在任务中指定加载其它目录中的依赖呢,具体场景如下:
1、我们开发了一个flink交换组件jar,该jar依赖了非常多第三方jar,如果合成一个shadejar得好几百MByte,所以我们把主jar和依赖jar拆开,把依赖jar预置到hdfs:///streampark/extlib;中。
2、在使用步骤1的主jar建立custom code任务时,允许用户指定yarn.provided.lib.dirs为hdfs:///streampark/extlib;
3、在使用其它的jar建立custom code任务r,或者flink sql任务时,不需要用户指定yarn.provided.lib.dirs,此时streampark会使用默认的yarn.provided.lib.dirs目录。
之所以我们没有把jar全部放到hdfs:///streampark/jars目录中,是为了避免在步骤3中的flink任务也去下载不必要的依赖,浪费网络和存储。
期待您的回复,谢谢!
GitHub link: https://github.com/apache/incubator-streampark/discussions/3743
----
This is an automatically sent email for dev@streampark.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampark.apache.org
Re: [D] flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs [incubator-streampark]
Posted by "StoneForests (via GitHub)" <gi...@apache.org>.
GitHub user StoneForests edited a comment on the discussion: flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs
> "/streampark/jars" stores the user's yarn-app mode public jar
The example in [Setting Job Parameters](https://streampark.apache.org/docs/get-started/how-to-use/#how-to-use-in-apache-streampark-1) shows yarn.provided.lib.dirs can be customized in yarn-application mode, but it does not work in fact...Is this a bug in StreamPark 2.1.4? Hope for your reply. THANK YOU!
GitHub link: https://github.com/apache/incubator-streampark/discussions/3743#discussioncomment-10248919
----
This is an automatically sent email for dev@streampark.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampark.apache.org
Re: [D] flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs [incubator-streampark]
Posted by "StoneForests (via GitHub)" <gi...@apache.org>.
GitHub user StoneForests edited a discussion: flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs
Java Version
java1.8
Scala Version
2.12
StreamPark Version
2.1.4
Flink Version
1.14.4
deploy mode
yarn-application
我参照streampark官网教程 [原生flink作业参数](https://streampark.apache.org/zh-CN/docs/user-guide/platformBasicUsage#%E5%8E%9F%E7%94%9Fflink%E4%BD%9C%E4%B8%9A%E5%8F%82%E6%95%B0) ,增加了两个动态参数

任务运行起来后,第一个自定义参数生效了,但第二个参数(yarn.provided.lib.dirs)没有生效

我看了flink的日志,其中yarn.provided.lib.dirs出现了两次,均为streampark的默认值,而非streampark界面上自定义值:


想请教一下:在yarn-application模式下,是不是无法自定义yarn.provided.lib.dirs的值?
`hdfs:///streampark/flink/flink-1.14.4/lib;hdfs:///streampark/jars;hdfs:///streampark/plugins;hdfs:///streampark/workspace/<appid>/lib;hdfs://zq-dev-cluster/streampark/shims/flink-1.14 `
如果真是如此,想问一下是否可以放开这个限制,允许yarn-application模式能在任务中指定加载其它目录中的依赖呢,具体场景如下:
1、我们开发了一个flink交换组件jar,该jar依赖了非常多第三方jar,全部整下来合计好几百MByte,我们把这些第三方jar都提前预置到hdfs:///streampark/extlib;中。
2、在使用步骤1的jar建立custom code任务时,允许用户指定yarn.provided.lib.dirs为hdfs:///streampark/extlib;
3、在使用其它的jar,或者flinksql任务时,使用streampark默认的yarn.provided.lib.dirs目录,从而避免去
之所以我们没有把jar全部放到hdfs:///streampark/jars目录中,是为了避免在步骤3中的fiinksql或其它jar的yarn container也去下载不必要的依赖,浪费网络和存储。
期待您的回复,谢谢!
GitHub link: https://github.com/apache/incubator-streampark/discussions/3743
----
This is an automatically sent email for dev@streampark.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampark.apache.org
Re: [D] flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs [incubator-streampark]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
GitHub user wolfboys added a comment to the discussion: flink yarn-application模式下,是否可以自定义每个flink任务的yarn.provided.lib.dirs
"/streampark/jars" stores the user's yarn-app mode public jar
GitHub link: https://github.com/apache/incubator-streampark/discussions/3743#discussioncomment-10186950
----
This is an automatically sent email for dev@streampark.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampark.apache.org