You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/03/08 10:44:07 UTC
[GitHub] [incubator-seatunnel] BenJFan opened a new issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
BenJFan opened a new issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
SparkLocalExample start failed. Report:
<img width="1405" alt="image" src="https://user-images.githubusercontent.com/32387433/157219713-dc6e571a-5e18-4625-9fde-f09666fd6c09.png">
Because dependency icebreg-spark-runtime have netty-common's class (package io.netty.util). Make Java can't choose right class.
#### icebreg-spark-runtime-0.13.1.jar
<img width="352" alt="image" src="https://user-images.githubusercontent.com/32387433/157220235-f88bf1f8-9c1a-4db4-a3ce-05d85957b19e.png">
#### netty-common-4.1.68.Final.jar
<img width="350" alt="image" src="https://user-images.githubusercontent.com/32387433/157221363-5ac1b4ec-17fe-4876-a07a-b9a355cec025.png">
So we should solve dependency conflict in icebreg side.
### SeaTunnel Version
dev
### SeaTunnel Config
```conf
-
```
### Running Command
```shell
-
```
### Error Exception
```log
Exception in thread "main" java.lang.NoSuchMethodError: io.netty.util.internal.ReflectionUtil.trySetAccessible(Ljava/lang/reflect/AccessibleObject;)Ljava/lang/Throwable;
at io.netty.channel.nio.NioEventLoop$5.run(NioEventLoop.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:210)
at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:149)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:127)
at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:60)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:49)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59)
at org.apache.spark.network.util.NettyUtils.createEventLoop(NettyUtils.java:50)
at org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:102)
at org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:99)
at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:71)
at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:461)
at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:57)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:249)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:424)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926)
at org.apache.seatunnel.spark.SparkEnvironment.prepare(SparkEnvironment.java:59)
at org.apache.seatunnel.spark.SparkEnvironment.prepare(SparkEnvironment.java:31)
at org.apache.seatunnel.config.ConfigBuilder.createEnv(ConfigBuilder.java:196)
at org.apache.seatunnel.config.ConfigBuilder.<init>(ConfigBuilder.java:66)
at org.apache.seatunnel.Seatunnel.entryPoint(Seatunnel.java:98)
at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:65)
at org.apache.seatunnel.example.spark.LocalSparkExample.main(LocalSparkExample.java:32)
```
### Flink or Spark Version
_No response_
### Java or Scala Version
_No response_
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ououtt commented on issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
ououtt commented on issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445#issuecomment-1061696117
@BenJFan @ZhangchengHu0923 I haven't run this main method directly on idea. I test it with script after packaging. It looks like you said it was caused by iceberg's netty-common package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ZhangchengHu0923 commented on issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
ZhangchengHu0923 commented on issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445#issuecomment-1061683388
> @ououtt Did you encounter this error while testing？
I debuged in local mode by sparkLocalMode recently,and it works well
@BenJFan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445#issuecomment-1061745748
> > @ououtt Did you encounter this error while testing？
> > I debuged in local mode by sparkLocalMode recently,and it works well
> > @BenJFan
@ZhangchengHu0923 Is your code up to date? If yes, I will check my environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ououtt commented on issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
ououtt commented on issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445#issuecomment-1062579054
Please pull the latest code and test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan closed issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
BenJFan closed issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445#issuecomment-1062585810
Fixed at #1439
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445#issuecomment-1062582876
> Please pull the latest code and test
Well done! I can run SparkLocalExample now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ZhangchengHu0923 edited a comment on issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
ZhangchengHu0923 edited a comment on issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445#issuecomment-1061758785
> > > @ououtt Did you encounter this error while testing？
> > > I debuged in local mode by sparkLocalMode recently,and it works well
> > > @BenJFan
>
> @ZhangchengHu0923 Is your code up to date? If yes, I will check my environment
I foked the code about 4 days ago @BenJFan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ZhangchengHu0923 commented on issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
ZhangchengHu0923 commented on issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445#issuecomment-1061706791
yes, i run it in idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ououtt commented on issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
ououtt commented on issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445#issuecomment-1061712038
I just submitted a PR and will fix the problem there. sorry for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] BenJFan commented on issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
BenJFan commented on issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445#issuecomment-1061644075
@ououtt Did you encounter this error while testing？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ZhangchengHu0923 commented on issue #1445: [Bug] [Core] SparkLocalExample can't run, netty util conflict.
Posted by GitBox <gi...@apache.org>.
ZhangchengHu0923 commented on issue #1445:
URL: https://github.com/apache/incubator-seatunnel/issues/1445#issuecomment-1061758785
> > > @ououtt Did you encounter this error while testing？
> > > I debuged in local mode by sparkLocalMode recently,and it works well
> > > @BenJFan
>
> @ZhangchengHu0923 Is your code up to date? If yes, I will check my environment
I foked the code about 4 days ago
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org