You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/08/31 06:25:29 UTC
[GitHub] [incubator-kyuubi] wangfeigithub opened a new issue, #3382: [Bug] Kyuubi failed to start Spark onYARN
wangfeigithub opened a new issue, #3382:
URL: https://github.com/apache/incubator-kyuubi/issues/3382
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
Kyuubi failed to start Spark onYARN ，Sometimes it's there and sometimes it's normal，
Zookeeper connection timed out
### Affects Version(s)
1.5.2
### Kyuubi Server Log Output
_No response_
### Kyuubi Engine Log Output
```logtalk
48066 [Driver] ERROR org.apache.kyuubi.shade.org.apache.curator.ConnectionState - Connection timed out for connection string (emr-header-1:2181,emr-header-2:2181,emr-worker-1:2181) and timeout (15000) / elapsed (31107)
org.apache.kyuubi.shade.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.kyuubi.shade.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.kyuubi.shade.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.kyuubi.shade.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:489)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
at org.apache.kyuubi.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:174)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient.registerService(ServiceDiscoveryClient.scala:99)
at org.apache.kyuubi.ha.client.ServiceDiscovery.start(ServiceDiscovery.scala:62)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.TFrontendService.start(TFrontendService.scala:74)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.Serverable.start(Serverable.scala:51)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.start(SparkSQLEngine.scala:58)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:163)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:143)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:143)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:201)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
48251 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
48351 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
48452 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
49553 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
49653 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
49754 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
50854 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
50955 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
51055 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
52156 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
52256 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
52357 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
53457 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
53558 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
53658 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
54759 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
54859 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
54960 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
56061 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
56161 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
56262 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
57362 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
57463 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
57563 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
58664 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
58764 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
58865 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
59965 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
60066 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
60167 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
61267 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
61368 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
61468 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
62569 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
62669 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
62770 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
63870 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
63971 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
64069 [Driver] ERROR org.apache.kyuubi.shade.org.apache.curator.ConnectionState - Connection timed out for connection string (emr-header-1:2181,emr-header-2:2181,emr-worker-1:2181) and timeout (15000) / elapsed (47110)
org.apache.kyuubi.shade.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.kyuubi.shade.org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:225)
at org.apache.kyuubi.shade.org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:94)
at org.apache.kyuubi.shade.org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:117)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:489)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
at org.apache.kyuubi.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:174)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient.registerService(ServiceDiscoveryClient.scala:99)
at org.apache.kyuubi.ha.client.ServiceDiscovery.start(ServiceDiscovery.scala:62)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.TFrontendService.start(TFrontendService.scala:74)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.Serverable.start(Serverable.scala:51)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.start(SparkSQLEngine.scala:58)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:163)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:143)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:143)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:201)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
64071 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
65172 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
65272 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
65373 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
66473 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
66574 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
66674 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
67775 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
67875 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
67976 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
69076 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
69176 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
69277 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
70378 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
70478 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
70579 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
71679 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
71780 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
71880 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
72981 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
73081 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
73182 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
74282 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
74383 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
74483 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
75584 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
75685 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
75785 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
76886 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
76986 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
77087 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
78187 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
78288 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
78388 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
79489 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
79590 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-1/172.16.3.238:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
79690 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
80072 [Driver] WARN org.apache.kyuubi.shade.org.apache.curator.ConnectionState - Connection attempt unsuccessful after 63113 (greater than max timeout of 60000). Resetting connection and trying again with a new connection.
80791 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - An exception was thrown while closing send thread for session 0x0.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
80897 [Driver] INFO org.apache.zookeeper.ZooKeeper - Session: 0x0 closed
80897 [Driver-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x0
80898 [Driver] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=emr-header-1:2181,emr-header-2:2181,emr-worker-1:2181 sessionTimeout=60000 watcher=org.apache.kyuubi.shade.org.apache.curator.ConnectionState@4bba1391
80898 [Driver] INFO org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes
80898 [Driver] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
80899 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-header-2/172.16.3.237:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
81000 [Driver-SendThread()] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever emr-worker-1/172.16.3.239:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1140)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1195)
81001 [Driver] ERROR org.apache.kyuubi.engine.spark.SparkTBinaryFrontendService - Error starting service EngineServiceDiscovery
org.apache.kyuubi.KyuubiException: Failed to create namespace '/kyuubi_1.5.2-incubating_USER_SPARK_SQL/itc_holo/default'
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:178)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient.registerService(ServiceDiscoveryClient.scala:99)
at org.apache.kyuubi.ha.client.ServiceDiscovery.start(ServiceDiscovery.scala:62)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.TFrontendService.start(TFrontendService.scala:74)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.Serverable.start(Serverable.scala:51)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.start(SparkSQLEngine.scala:58)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:163)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:143)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:143)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:201)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /kyuubi_1.5.2-incubating_USER_SPARK_SQL/itc_holo/default
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1734)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
at org.apache.kyuubi.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:174)
... 28 more
81003 [Driver] INFO org.apache.kyuubi.engine.spark.SparkTBinaryFrontendService - SparkTBinaryFrontend has stopped
81003 [Driver] ERROR org.apache.kyuubi.engine.spark.SparkSQLEngine - Error starting service SparkTBinaryFrontend
org.apache.kyuubi.KyuubiException: Failed to Start SparkTBinaryFrontend
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:52)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.TFrontendService.start(TFrontendService.scala:74)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1$adapted(CompositeService.scala:45)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.start(CompositeService.scala:45)
at org.apache.kyuubi.service.Serverable.start(Serverable.scala:51)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.start(SparkSQLEngine.scala:58)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:163)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:143)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:143)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:201)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:737)
Caused by: org.apache.kyuubi.KyuubiException: Failed to create namespace '/kyuubi_1.5.2-incubating_USER_SPARK_SQL/itc_holo/default'
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:178)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient.registerService(ServiceDiscoveryClient.scala:99)
at org.apache.kyuubi.ha.client.ServiceDiscovery.start(ServiceDiscovery.scala:62)
at org.apache.kyuubi.service.CompositeService.$anonfun$start$1(CompositeService.scala:47)
... 25 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /kyuubi_1.5.2-incubating_USER_SPARK_SQL/itc_holo/default
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1734)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
at org.apache.kyuubi.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
at org.apache.kyuubi.shade.org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at org.apache.kyuubi.ha.client.zookeeper.ServiceDiscoveryClient$.createServiceNode(ServiceDiscoveryClient.scala:174)
... 28 more
```
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wangfeigithub commented on issue #3382: [Bug] Kyuubi failed to start Spark onYARN
Posted by GitBox <gi...@apache.org>.
wangfeigithub commented on issue #3382:
URL: https://github.com/apache/incubator-kyuubi/issues/3382#issuecomment-1235048581
I wonder why it occasionally happens.-------------------
java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC ---Could this mistake be the root cause? What causes it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wangfeigithub commented on issue #3382: [Bug] Kyuubi failed to start Spark onYARN
Posted by GitBox <gi...@apache.org>.
wangfeigithub commented on issue #3382:
URL: https://github.com/apache/incubator-kyuubi/issues/3382#issuecomment-1234983470
Sometimes there are mistakes, sometimes there are no mistakes. And I changed another Zookeeper also occasionally appeared like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wangfeigithub commented on issue #3382: [Bug] Kyuubi failed to start Spark onYARN
Posted by GitBox <gi...@apache.org>.
wangfeigithub commented on issue #3382:
URL: https://github.com/apache/incubator-kyuubi/issues/3382#issuecomment-1235047859
>
Zk has no error and I tried two ZKS. The error on the YARN page above will occasionally occur. Nor should the Internet. Using Ali's EMR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wangfeigithub commented on issue #3382: [Bug] Kyuubi failed to start Spark onYARN
Posted by GitBox <gi...@apache.org>.
wangfeigithub commented on issue #3382:
URL: https://github.com/apache/incubator-kyuubi/issues/3382#issuecomment-1235041381
Container from a bad node Exit code :50 is displayed when the SQL is executed. So what does this look like
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #3382: [Bug] Kyuubi failed to start Spark onYARN
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #3382:
URL: https://github.com/apache/incubator-kyuubi/issues/3382#issuecomment-1232640677
Your error code indicates that the root cause is "Failed to create namespace '/kyuubi_1.5.2-incubating_USER_SPARK_SQL/itc_holo/default'", please check the Zookeeper log what happens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #3382: [Bug] Kyuubi failed to start Spark onYARN
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #3382:
URL: https://github.com/apache/incubator-kyuubi/issues/3382#issuecomment-1235042925
> Sometimes there are mistakes, sometimes there are no mistakes. And I changed another Zookeeper also occasionally appeared like this
what's the error stacktrace? the client side error indicates it's a network issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wangfeigithub closed issue #3382: [Bug] Kyuubi failed to start Spark onYARN
Posted by GitBox <gi...@apache.org>.
wangfeigithub closed issue #3382: [Bug] Kyuubi failed to start Spark onYARN
URL: https://github.com/apache/incubator-kyuubi/issues/3382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #3382: [Bug] Kyuubi failed to start Spark onYARN
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #3382:
URL: https://github.com/apache/incubator-kyuubi/issues/3382#issuecomment-1235049223
> java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
I think it was caused by your Spark application classpath jars conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] github-actions[bot] commented on issue #3382: [Bug] Kyuubi failed to start Spark onYARN
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #3382:
URL: https://github.com/apache/incubator-kyuubi/issues/3382#issuecomment-1232515298
Hello @wangfeigithub,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi (Incubating).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on issue #3382: [Bug] Kyuubi failed to start Spark onYARN
Posted by GitBox <gi...@apache.org>.
pan3793 commented on issue #3382:
URL: https://github.com/apache/incubator-kyuubi/issues/3382#issuecomment-1235042668
> Container from a bad node Exit code :50 is displayed when the SQL is executed. So what does this look like
sounds like another issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] wangfeigithub commented on issue #3382: [Bug] Kyuubi failed to start Spark onYARN
Posted by GitBox <gi...@apache.org>.
wangfeigithub commented on issue #3382:
URL: https://github.com/apache/incubator-kyuubi/issues/3382#issuecomment-1235100736
> > java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
>
> I think it was caused by your Spark application classpath jars conflicts.
Maybe there is a conflict with org.slf4j. I will try to remove it to see the Zookeeper problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org