You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by "king-sjr-007 (via GitHub)" <gi...@apache.org> on 2023/09/21 09:27:00 UTC
[GitHub] [incubator-streampark] king-sjr-007 opened a new issue, #3159: [Bug] application mode start flink job. provided.lib.dirs error
king-sjr-007 opened a new issue, #3159:
URL: https://github.com/apache/incubator-streampark/issues/3159
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-streampark/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### Java Version
1.8
### Scala Version
2.12.x
### StreamPark Version
2.1.1
### Flink Version
1.16.2
### deploy mode
yarn-application
### What happened
with submit a flink job in yarn-application mode. Encountered a situation where the error message is Caused by: java.lang.IllegalArgumentException: The "yarn.provided.lib.dirs" should only contain dirs accessible from all worker nodes, while the "file:/streampark/flink/flink-1.16.2/lib" is local.
at org.apache.flink.yarn.Utils.getRemoteSharedLibPaths(Utils.java:662)
at org.apache.flink.yarn.Utils.getQualifiedRemoteProvidedLibDirs(Utils.java:641)
at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:814)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:623)
at org.apache.flink.yarn.YarnClusterDescriptor.deployApplicationCluster(YarnClusterDescriptor.java:471)
... 28 more
But Whether it is yarn-site or flink-conf, yarn.provided.lib.dir=hdfs://XXX has been configured.
And in streampark.out yarn.provided.lib.dir appears multiple times, the first few times it was hdfs://XXX ,Only the last occurrence of yarn.provided.lib.dir is file:/streampark/flink/flink-1.16.2/lib,is so wired.
### Error Exception
_No response_
### Screenshots
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!(您是否要贡献这个PR?)
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] application mode start flink job. provided.lib.dirs error [incubator-streampark]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys closed issue #3159: [Bug] application mode start flink job. provided.lib.dirs error
URL: https://github.com/apache/incubator-streampark/issues/3159
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] application mode start flink job. provided.lib.dirs error [incubator-streampark]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on issue #3159:
URL: https://github.com/apache/incubator-streampark/issues/3159#issuecomment-1844373264
Please check the configuration item in the application.yml: `streampark.workspace.remote`: It needs to be set to the path of HDFS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org