You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/06/17 08:25:28 UTC
[GitHub] [incubator-kyuubi] waywtdcc opened a new issue, #2904: [Bug] Classnotfound org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
waywtdcc opened a new issue, #2904:
URL: https://github.com/apache/incubator-kyuubi/issues/2904
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
package:
./build/dist --tgz --spark-provided --flink-provided --mvn /opt/apps/soft/apache-maven-3.8.3/bin/mvn -Dflink.version=1.14.4
### Affects Version(s)
1.6.0
### Kyuubi Server Log Output
```logtalk
2022-06-17 16:13:05.998 INFO org.apache.kyuubi.util.ThreadUtils: KyuubiSessionManager-exec-pool: pool size: 100, wait queue size: 100, thread keepalive time: 60000 ms
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/javax/ws/rs/core/NoContentException
at org.apache.hadoop.yarn.util.timeline.TimelineUtils.<clinit>(TimelineUtils.java:60)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:200)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.kyuubi.engine.YarnApplicationOperation.initialize(YarnApplicationOperation.scala:37)
at org.apache.kyuubi.engine.KyuubiApplicationManager.$anonfun$initialize$1(KyuubiApplicationManager.scala:41)
at org.apache.kyuubi.engine.KyuubiApplicationManager.$anonfun$initialize$1$adapted(KyuubiApplicationManager.scala:39)
at scala.collection.immutable.Stream.foreach(Stream.scala:533)
at org.apache.kyuubi.engine.KyuubiApplicationManager.initialize(KyuubiApplicationManager.scala:39)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
at org.apache.kyuubi.session.SessionManager.initialize(SessionManager.scala:268)
at org.apache.kyuubi.session.KyuubiSessionManager.initialize(KyuubiSessionManager.scala:57)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
at org.apache.kyuubi.service.AbstractBackendService.initialize(AbstractBackendService.scala:209)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:46)
at org.apache.kyuubi.server.KyuubiServer.initialize(KyuubiServer.scala:140)
at org.apache.kyuubi.server.KyuubiServer$.startServer(KyuubiServer.scala:60)
at org.apache.kyuubi.server.KyuubiServer$.main(KyuubiServer.scala:97)
at org.apache.kyuubi.server.KyuubiServer.main(KyuubiServer.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
```
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #2904: [Bug] Classnotfound org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2904:
URL: https://github.com/apache/incubator-kyuubi/issues/2904#issuecomment-1159558375
I raise the priority to blocker, we may need to revert https://github.com/apache/incubator-kyuubi/pull/2003 if Hadoop 3.3.4 is not available before Kyuubi 1.6.0 release because of HADOOP-18033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] limboys commented on issue #2904: [Bug] ClassNotFound org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
Posted by "limboys (via GitHub)" <gi...@apache.org>.
limboys commented on issue #2904:
URL: https://github.com/apache/kyuubi/issues/2904#issuecomment-1415213895
This issue reappears in versions of Hadoop 3.3.4 and Spark-3.1-bin-hadoop3 when executing
```
base) [clvprd@hadoop-node3 jars]$ spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1G --num-executors 3 --executor-memory 1G --executor-cores 1 /app/bigdata/hadoop/server/spark-3.3.1-bin-hadoop3/examples/jars/spark-examples_2.12-3.3.1.jar 100
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/javax/ws/rs/core/NoContentException
at org.apache.hadoop.yarn.util.timeline.TimelineUtils.<clinit>(TimelineUtils.java:60)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:200)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:191)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1327)
at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1764)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
23/02/03 14:44:57 INFO ShutdownHookManager: Shutdown hook called
23/02/03 14:44:57 INFO ShutdownHookManager: Deleting directory /tmp/spark-bed55fe1-7af2-42e2-a246-24f222abcdae
```
when the spark version is changed to spark-3.1-bin-hadoop3 is ok
```
spark-submit \
> --class org.apache.spark.examples.SparkPi \
> --master yarn \
> --deploy-mode cluster \
> --driver-memory 1G \
> --num-executors 3 \
> --executor-memory 1G \
> --executor-cores 1 \
> /app/bigdata/hadoop/server/spark-3.2.3-bin-hadoop3.2/examples/jars/spark-examples_2.12-3.2.3.jar 100
2023-02-03 15:16:06,251 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2023-02-03 15:16:06,307 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at hadoop-node1/10.82.97.120:8032
2023-02-03 15:16:06,465 INFO client.AHSProxy: Connecting to Application History server at hadoop-node1/10.82.97.120:10200
2023-02-03 15:16:06,584 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
2023-02-03 15:16:07,144 INFO conf.Configuration: resource-types.xml not found
2023-02-03 15:16:07,144 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2023-02-03 15:16:07,161 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (4096 MB per container)
2023-02-03 15:16:07,161 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
2023-02-03 15:16:07,162 INFO yarn.Client: Setting up container launch context for our AM
2023-02-03 15:16:07,162 INFO yarn.Client: Setting up the launch environment for our AM container
2023-02-03 15:16:07,174 INFO yarn.Client: Preparing resources for our AM container
....
2023-02-03 15:16:46,007 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: hadoop-node2
ApplicationMaster RPC port: 43052
queue: default
start time: 1675408575548
final status: **SUCCEEDED**
tracking URL: http://hadoop-node1:8088/proxy/application_1675404773406_0001/
user: clvprd
2023-02-03 15:16:46,020 INFO util.ShutdownHookManager: Shutdown hook called
2023-02-03 15:16:46,021 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-fc581b74-8986-4937-9d2c-62258c7c1c31
2023-02-03 15:16:46,025 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-9cb0db4d-0932-43ab-bd2a-e08c1c4e6df2
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on issue #2904: [Bug] ClassNotFound org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #2904:
URL: https://github.com/apache/kyuubi/issues/2904#issuecomment-1415225395
@limboys spark-3.3.1-bin-hadoop3 ships Hadoop 3.3.2[1], have you replaced the Hadoop jars to 3.3.4?
[1] https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.12/3.3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] sbhatm1213 commented on issue #2904: [Bug] ClassNotFound org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
Posted by GitBox <gi...@apache.org>.
sbhatm1213 commented on issue #2904:
URL: https://github.com/apache/incubator-kyuubi/issues/2904#issuecomment-1315796372
With Hadoop 3.1.1 I was facing this error
`Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException`
I upgraded to Hadoop 3.3.3 , still getting the error.
Only if I disable the yarn-timeline-service, I'm able to run the job without the error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #2904: [Bug] ClassNotFound org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2904:
URL: https://github.com/apache/incubator-kyuubi/issues/2904#issuecomment-1316297744
this issue was introduced in 3.3.2, and got fixed in 3.3.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #2904: [Bug] Classnotfound org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #2904:
URL: https://github.com/apache/incubator-kyuubi/issues/2904#issuecomment-1158638940
May caused by HADOOP-18033, you can replace the Hadoop 3.3.2 jars w/ 3.3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #2904: [Bug] Classnotfound org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #2904:
URL: https://github.com/apache/incubator-kyuubi/issues/2904#issuecomment-1158643036
yarn.timeline-service.enabled is set to true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 closed issue #2904: [Bug] ClassNotFound org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
Posted by GitBox <gi...@apache.org>.
pan3793 closed issue #2904: [Bug] ClassNotFound org.apache.hadoop.shaded.javax.ws.rs.core.NoContentException
URL: https://github.com/apache/incubator-kyuubi/issues/2904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org