You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by GitBox <gi...@apache.org> on 2022/04/08 09:01:46 UTC
[GitHub] [incubator-seatunnel] arenaswan opened a new issue, #1676: [Bug] [HDFS] java.lang.IllegalAccessError
arenaswan opened a new issue, #1676:
URL: https://github.com/apache/incubator-seatunnel/issues/1676
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues.
### What happened
When I use the version 1.5.7 of seatunnel,it worked perfect. But when I upgrade it to 2.1.0, it report an error.
### SeaTunnel Version
2.1.0
### SeaTunnel Config
```conf
env {
spark.app.name = "seatunnel"
spark.executor.instances = 2
spark.executor.cores = 5
spark.executor.memory = "4g"
spark.sql.catalogImplementation = "hive"
}
source {
hive {
pre_sql = "select * from tmp.test_tmp"
table_name = "test_tmp"
}
}
transform {
}
sink {
clickhouse {
host = "IP:PORT"
database = "backup"
bulk_size = 200000
table = "test_tmp"
fields = ["title","name"]
username = "default"
password = "****************"
}
}
```
### Running Command
```shell
./bin/start-seatunnel-spark.sh --master yarn --deploy-mode cluster --config ./config/test.conf
```
### Error Exception
```log
[2022-04-08 16:19:20.712]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
.scala:248)
at scala.concurrent.Promise$class.tryFailure(Promise.scala:112)
at scala.concurrent.impl.Promise$DefaultPromise.tryFailure(Promise.scala:157)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:686)
Caused by: java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:3202)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3247)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3286)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3305)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1869)
at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:74)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:531)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at org.apache.seatunnel.spark.SparkEnvironment.prepare(SparkEnvironment.java:59)
at org.apache.seatunnel.spark.SparkEnvironment.prepare(SparkEnvironment.java:31)
at org.apache.seatunnel.config.ConfigBuilder.createEnv(ConfigBuilder.java:196)
at org.apache.seatunnel.config.ConfigBuilder.<init>(ConfigBuilder.java:66)
at org.apache.seatunnel.Seatunnel.entryPoint(Seatunnel.java:98)
at org.apache.seatunnel.Seatunnel.run(Seatunnel.java:65)
at org.apache.seatunnel.SeatunnelSpark.main(SeatunnelSpark.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:667)
```
### Flink or Spark Version
spark 2.4.0
### Java or Scala Version
jdk1.8
### Screenshots
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] userkdg commented on issue #1676: [Bug] [HDFS] java.lang.IllegalAccessError
Posted by "userkdg (via GitHub)" <gi...@apache.org>.
userkdg commented on issue #1676:
URL: https://github.com/apache/seatunnel/issues/1676#issuecomment-1600028503
> 
>
> ```
> <artifactSet>
> <excludes>
> <exclude>org.apache.hadoop:hadoop-hdfs</exclude>
> <exclude>org.apache.flink:flink-shaded-hadoop-2</exclude>
> </excludes>
> </artifactSet>
> ```
>
> 修改root pom.xml后，我可以成功运行示例
可以提供一下你的`seatunnel-connector-spark-hbase-xxx.jar`包？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] someorz commented on issue #1676: [Bug] [HDFS] java.lang.IllegalAccessError
Posted by GitBox <gi...@apache.org>.
someorz commented on issue #1676:
URL: https://github.com/apache/incubator-seatunnel/issues/1676#issuecomment-1121850902
```
<artifactSet>
<excludes>
<exclude>org.apache.hadoop:hadoop-hdfs</exclude>
<exclude>org.apache.flink:flink-shaded-hadoop-2</exclude>
</excludes>
</artifactSet>
```
after modify root pom.xml,i can run examples successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] ruanwenjun commented on issue #1676: [Bug] [HDFS] java.lang.IllegalAccessError
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #1676:
URL: https://github.com/apache/incubator-seatunnel/issues/1676#issuecomment-1092666803
It seems this is caused by the hadoop version mismatch, I find an answer at https://stackoverflow.com/questions/62880009/error-through-remote-spark-job-java-lang-illegalaccesserror-class-org-apache-h
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] xinda77 commented on issue #1676: [Bug] [HDFS] java.lang.IllegalAccessError
Posted by GitBox <gi...@apache.org>.
xinda77 commented on issue #1676:
URL: https://github.com/apache/incubator-seatunnel/issues/1676#issuecomment-1107832083
i have got the same problem ，follow the above method, but it doesn't work. and search the maven dependency tree doesn't find any the version of 2.7.x hadoop jars .i have no idea. my hadoop version is 3.0.0-cdh6.3.2
error stack
[2022-04-24 20:16:19.413]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
.tryFailure(Promise.scala:157)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:692)
Caused by: java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-seatunnel] zhfish commented on issue #1676: [Bug] [HDFS] java.lang.IllegalAccessError
Posted by GitBox <gi...@apache.org>.
zhfish commented on issue #1676:
URL: https://github.com/apache/incubator-seatunnel/issues/1676#issuecomment-1111942786
marked,
i have got the same problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [seatunnel] userkdg commented on issue #1676: [Bug] [HDFS] java.lang.IllegalAccessError
Posted by "userkdg (via GitHub)" <gi...@apache.org>.
userkdg commented on issue #1676:
URL: https://github.com/apache/seatunnel/issues/1676#issuecomment-1600028068
up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org