You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@streampark.apache.org by "pj001 (via GitHub)" <gi...@apache.org> on 2023/12/18 09:38:53 UTC
[I] streampark commons-cli NoSuchMethodError [incubator-streampark]
pj001 opened a new issue, #3408:
URL: https://github.com/apache/incubator-streampark/issues/3408
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-streampark/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### Java Version
1.8
### Scala Version
2.11.x
### StreamPark Version
2.1.2
### Flink Version
1.13.6
### deploy mode
yarn-application
### What happened
streampark 使用 flink1.13.6 （commons-cli-1.3.1）提交任务，出现 commons-cli 冲突，我这边是首次提交无问题，某次重启任务时此问题出现：
1.检查项目是否打包依赖，排除commons-cli打包后提交任务。问题未解决
2.尝试将streampark lib 目录包更换与flink对应版本一致，重启streampark。问题未解决。
3.尝试替换streampark hdfs flink 与本地备份的commons-cli jar,重启streampark。问题未解决。
4.删除本地streampark workspace，重启集群，启动任务，问题未解决。
5.删除hdfs streampark workspace,重启集群，启动任务，问题未解决。
6.删除本地streampark workspace，hdfs streampark workspace,替换 flink1.13 streampark2.1.2 commons-cli 为 commons-cli-1.3.1 问题解决。
7.回滚streampark2.1.1,问题解决。
猜测可能是streampark重启任务任务时从本地或者hdfs的备份加载flink依赖导致失败，或者是加载了整个flink 任务（flink项目代码）的 jar ，导致冲突。
### Error Exception
```log
java.util.concurrent.CompletionException: java.lang.reflect.InvocationTargetException
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.streampark.flink.client.FlinkClient$.$anonfun$proxy$1(FlinkClient.scala:80)
at org.apache.streampark.flink.proxy.FlinkShimsProxy$.$anonfun$proxy$1(FlinkShimsProxy.scala:60)
at org.apache.streampark.common.util.ClassLoaderUtils$.runAsClassLoader(ClassLoaderUtils.scala:38)
at org.apache.streampark.flink.proxy.FlinkShimsProxy$.proxy(FlinkShimsProxy.scala:60)
at org.apache.streampark.flink.client.FlinkClient$.proxy(FlinkClient.scala:75)
at org.apache.streampark.flink.client.FlinkClient$.submit(FlinkClient.scala:49)
at org.apache.streampark.flink.client.FlinkClient.submit(FlinkClient.scala)
at org.apache.streampark.console.core.service.impl.ApplicationServiceImpl.lambda$start$10(ApplicationServiceImpl.java:1566)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
... 3 more
Caused by: java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder;
at org.apache.flink.client.cli.DynamicPropertiesUtil.<clinit>(DynamicPropertiesUtil.java:39)
at org.apache.flink.client.cli.GenericCLI.addGeneralOptions(GenericCLI.java:108)
at org.apache.flink.client.cli.CliFrontend.<init>(CliFrontend.java:136)
at org.apache.flink.client.cli.CliFrontend.<init>(CliFrontend.java:119)
at org.apache.streampark.flink.client.trait.FlinkClientTrait.getCustomCommandLines(FlinkClientTrait.scala:293)
at org.apache.streampark.flink.client.trait.FlinkClientTrait.getCommandLineOptions(FlinkClientTrait.scala:379)
at org.apache.streampark.flink.client.trait.FlinkClientTrait.getCommandLineOptions$(FlinkClientTrait.scala:378)
at org.apache.streampark.flink.client.impl.YarnApplicationClient$.getCommandLineOptions(YarnApplicationClient.scala:44)
at org.apache.streampark.flink.client.trait.FlinkClientTrait.getCommandLineAndFlinkConfig(FlinkClientTrait.scala:309)
at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit(FlinkClientTrait.scala:81)
at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit$(FlinkClientTrait.scala:60)
at org.apache.streampark.flink.client.impl.YarnApplicationClient$.submit(YarnApplicationClient.scala:44)
at org.apache.streampark.flink.client.FlinkClientHandler$.submit(FlinkClientHandler.scala:40)
at org.apache.streampark.flink.client.FlinkClientHandler.submit(FlinkClientHandler.scala)
... 16 more
```
### Screenshots
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!(您是否要贡献这个PR?)
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] streampark commons-cli NoSuchMethodError [incubator-streampark]
Posted by "sunderdont (via GitHub)" <gi...@apache.org>.
sunderdont commented on issue #3408:
URL: https://github.com/apache/incubator-streampark/issues/3408#issuecomment-2219418018
commons-cli-1.2 in flink-shaded-hadoop-3-uber-xxx.jar is lower than streampark-2.1.2's commons-cli-1.5.0，who is short of org.apache.commons.cli.Option$Builder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] streampark commons-cli NoSuchMethodError [incubator-streampark]
Posted by "pj001 (via GitHub)" <gi...@apache.org>.
pj001 commented on issue #3408:
URL: https://github.com/apache/incubator-streampark/issues/3408#issuecomment-1862012660
根据实测，streampark2.1.2 与此包存在冲突。
<img width="461" alt="46773a259f1c9ceb95a18a3c08abd8d" src="https://github.com/apache/incubator-streampark/assets/39849195/bee0b863-456b-43d7-aedb-8d63565fecd8">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] streampark commons-cli NoSuchMethodError [incubator-streampark]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on issue #3408:
URL: https://github.com/apache/incubator-streampark/issues/3408#issuecomment-1867788722
You can check if there are common-cli conflicts in hadoop-related packages. e.g flink-shaded-hadoop-x-uber.jar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] streampark commons-cli NoSuchMethodError [incubator-streampark]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys closed issue #3408: streampark commons-cli NoSuchMethodError
URL: https://github.com/apache/incubator-streampark/issues/3408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@streampark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org