You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "inkkim (via GitHub)" <gi...@apache.org> on 2024/11/08 01:20:11 UTC
[I] [Bug] Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress() [kyuubi]
inkkim opened a new issue, #6799:
URL: https://github.com/apache/kyuubi/issues/6799
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:174)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.kyuubi.KyuubiException: Failed to initialize SparkSQLEngine: org.apache.kyuubi.util.JavaUtils.findLocalInetAddress()Ljava/net/InetAddress;
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:357)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:342)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:342)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
### Affects Version(s)
v1.10
### Kyuubi Server Log Output
```logtalk
2024-11-06 18:35:04.034 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: Starting
2024-11-06 18:35:04.034 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Initiating client connection, connectString=ZOOKE
EPER-001:2181,ZOOKEEPER-002:2181,ZOOKEEPER-003:2181 sessionTimeout=60000 watcher=org.apache.kyuubi.shaded.curator.ConnectionState@1e19cc16
2024-11-06 18:35:04.035 INFO KyuubiSessionManager-exec-pool: Thread-219-SendThread(ZOOKEEPER-003:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Opening socket co
nnection to server ZOOKEEPER-003/10.10.10.10:2181. Will not attempt to authenticate using SASL (unknown error)
2024-11-06 18:35:04.036 INFO KyuubiSessionManager-exec-pool: Thread-219-SendThread(ZOOKEEPER-003:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Socket connection
established to ZOOKEEPER-003/10.10.10.10:2181, initiating session
2024-11-06 18:35:04.037 INFO KyuubiSessionManager-exec-pool: Thread-219-SendThread(ZOOKEEPER-003:2181) org.apache.kyuubi.shaded.zookeeper.ClientCnxn: Session establish
ment complete on server ZOOKEEPER-003/10.10.10.10:2181, sessionid = 0x300231d79b100b7, negotiated timeout = 40000
2024-11-06 18:35:04.038 INFO KyuubiSessionManager-exec-pool: Thread-219-EventThread org.apache.kyuubi.shaded.curator.framework.state.ConnectionStateManager: State change: CONNECTED
2024-11-06 18:35:04.044 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.Utils: Loading Kyuubi properties from /usr/local/spark/conf/spark-defaults.conf
2024-11-06 18:35:04.046 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.engine.ProcBuilder: Logging to /opt/apache-kyuubi-1.10.0-bin/work/kyuubi/kyuubi-spark-sql
-engine.log.10
2024-11-06 18:35:04.047 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.engine.EngineRef: Launching engine:
/opt/spark-3.5.1-bin-hadoop3/bin/spark-submit \
--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
--conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
--conf spark.kyuubi.client.ipAddress=127.0.0.1 \
--conf spark.kyuubi.client.version=1.10.0 \
--conf spark.kyuubi.engine.appMgrInfo=eyJyZXNvdXJjZU1hbmFnZXIiOiJ5YXJuIiwia3ViZXJuZXRlc0luZm8iOnsiY29udGV4dCI6bnVsbCwibmFtZXNwYWNlIjpudWxsfX0= \
--conf spark.kyuubi.engine.engineLog.path=/opt/apache-kyuubi-1.10.0-bin/work/kyuubi/kyuubi-spark-sql-engine.log.10 \
--conf spark.kyuubi.engine.share.level=USER \
--conf spark.kyuubi.engine.submit.time=1730885704044 \
--conf spark.kyuubi.engine.type=SPARK_SQL \
--conf spark.kyuubi.engine.ui.stop.enabled=true \
--conf spark.kyuubi.frontend.protocols=THRIFT_BINARY,REST \
--conf spark.kyuubi.ha.addresses=ZOOKEEPER-001:2181,ZOOKEEPER-002:2181,ZOOKEEPER-003:2181 \
--conf spark.kyuubi.ha.engine.ref.id=66be7e12-49dc-4de1-890f-297c086342a8 \
--conf spark.kyuubi.ha.namespace=/kyuubi_1.10.0_USER_SPARK_SQL/kyuubi/default \
--conf spark.kyuubi.operation.result.format=arrow \
--conf spark.kyuubi.server.ipAddress=10.10.10.10 \
--conf spark.kyuubi.session.connection.url=10.10.10.10:10009 \
--conf spark.kyuubi.session.engine.idle.timeout=PT0M \
--conf spark.kyuubi.session.engine.spark.showProgress=true \
--conf spark.kyuubi.session.real.user=kyuubi \
--conf spark.app.name=kyuubi_USER_SPARK_SQL_kyuubi_default_66be7e12-49dc-4de1-890f-297c086342a8 \
--conf spark.driver.extraJavaOptions=-Daws.region=us-west-2 \
--conf spark.executor.extraJavaOptions=-Daws.region=us-west-2 \
--conf spark.sql.catalog.sample=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.sample.io-impl=org.apache.iceberg.aws.s3.S3FileIO \
--conf spark.sql.catalog.sample.s3.access-key-id=sample \
--conf spark.sql.catalog.sample.s3.endpoint=http://10.10.10.10:9020 \
--conf spark.sql.catalog.sample.s3.path-style-access=true \
--conf spark.sql.catalog.sample.s3.secret-access-key=********************** \
--conf spark.sql.catalog.sample.type=rest \
--conf spark.sql.catalog.sample.uri=http://rest.test.com:9083 \
--conf spark.sql.catalog.sample.warehouse=s3a://sample/ \
--conf spark.sql.defaultCatalog=sample \
--conf spark.yarn.maxAppAttempts=1 \
--conf spark.yarn.tags=KYUUBI,66be7e12-49dc-4de1-890f-297c086342a8 \
--proxy-user kyuubi /opt/apache-kyuubi-1.10.0-bin/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.10.0.jar
2024-11-06 18:35:39.450 INFO Curator-Framework-0 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
2024-11-06 18:35:39.452 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Session: 0x300231d79b100b7 closed
2024-11-06 18:35:39.452 INFO KyuubiSessionManager-exec-pool: Thread-219-EventThread org.apache.kyuubi.shaded.zookeeper.ClientCnxn: EventThread shut down for session: 0x300231d79b100b7
2024-11-06 18:35:39.453 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.operation.LaunchEngine: Processing kyuubi's query[c4c29906-fd8b-47d5-8228-b18dcf6a67d6]: RUNNING_STATE -> ERROR_STATE, time taken: 35.419 seconds
2024-11-06 18:35:39.453 INFO KyuubiSessionManager-exec-pool: Thread-219 org.apache.kyuubi.operation.OperationAuditLogger: operation=c4c29906-fd8b-47d5-8228-b18dcf6a67d6 opType=LaunchEngine state=ERROR user=kyuubi session=66be7e12-49dc-4de1-890f-297c086342a8
2024-11-06 18:35:39.741 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-206 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [66be7e12-49dc-4de1-890f-297c086342a8]
2024-11-06 18:35:39.741 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-206 org.apache.kyuubi.session.KyuubiSessionManager: kyuubi's KyuubiSessionImpl with SessionHandle [66be7e12-49dc-4de1-890f-297c086342a8] is closed, current opening sessions 0
2024-11-06 18:35:39.741 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-206 org.apache.kyuubi.operation.LaunchEngine: Processing kyuubi's query[c4c29906-fd8b-47d5-8228-b18dcf6a67d6]: ERROR_STATE -> CLOSED_STATE, time taken: 35.707 seconds
2024-11-06 18:35:39.741 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-206 org.apache.kyuubi.operation.OperationAuditLogger: operation=c4c29906-fd8b-47d5-8228-b18dcf6a67d6 opType=LaunchEngine state=CLOSED user=kyuubi session=66be7e12-49dc-4de1-890f-297c086342a8
2024-11-06 18:35:39.742 INFO KyuubiTBinaryFrontendHandler-Pool: Thread-206 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [66be7e12-49dc-4de1-890f-297c086342a8]
2024-11-06 18:36:44.329 ERROR SIGTERM handler org.apache.kyuubi.server.KyuubiServer: RECEIVED SIGNAL 15: TERM
2024-11-06 18:36:44.332 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiRestFrontendService] is stopping.
2024-11-06 18:36:44.335 INFO shutdown-hook-0 org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@34780cd9{HTTP/1.1, (http/1.1)}{10.10.10.10:10099}
2024-11-06 18:36:44.337 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiRestFrontendService: Service[KyuubiRestFrontendService] is stopped.
2024-11-06 18:36:44.337 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiTBinaryFrontend] is stopping.
2024-11-06 18:36:44.337 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Service: [KyuubiServiceDiscovery] is stopping.
2024-11-06 18:36:44.339 INFO Curator-Framework-0 org.apache.kyuubi.shaded.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
2024-11-06 18:36:44.339 WARN main-EventThread org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: This Kyuubi instance 10.10.10.10:10009 is now de-registered from ZooKeeper. The server will be shut down after the last client session completes.
2024-11-06 18:36:44.340 INFO shutdown-hook-0 org.apache.kyuubi.shaded.zookeeper.ZooKeeper: Session: 0x300231d79b1009c closed
2024-11-06 18:36:44.340 INFO shutdown-hook-0 org.apache.kyuubi.ha.client.KyuubiServiceDiscovery: Service[KyuubiServiceDiscovery] is stopped.
2024-11-06 18:36:44.340 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Service[KyuubiTBinaryFrontend] is stopped.
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiTBinaryFrontendService: KyuubiTBinaryFrontend has stopped
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiBackendService] is stopping.
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [KyuubiSessionManager] is stopping.
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service: [KyuubiOperationManager] is stopping.
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.operation.KyuubiOperationManager: Service[KyuubiOperationManager] is stopped.
2024-11-06 18:36:44.341 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service: [MetadataManager] is stopping.
2024-11-06 18:36:44.342 INFO shutdown-hook-0 com.zaxxer.hikari.HikariDataSource: jdbc-metadata-store-pool - Shutdown initiated...
2024-11-06 18:36:44.344 INFO shutdown-hook-0 com.zaxxer.hikari.HikariDataSource: jdbc-metadata-store-pool - Shutdown completed.
2024-11-06 18:36:44.344 INFO shutdown-hook-0 org.apache.kyuubi.server.metadata.MetadataManager: Service[MetadataManager] is stopped.
2024-11-06 18:36:44.344 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service: [TempFileService] is stopping.
2024-11-06 18:36:44.345 INFO shutdown-hook-0 org.apache.kyuubi.service.TempFileService: Service[TempFileService] is stopped.
2024-11-06 18:36:44.345 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service: [HadoopCredentialsManager] is stopping.
2024-11-06 18:36:44.346 INFO shutdown-hook-0 org.apache.kyuubi.credentials.HadoopCredentialsManager: Service[HadoopCredentialsManager] is stopped.
2024-11-06 18:36:44.346 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service: [KyuubiApplicationManager] is stopping.
2024-11-06 18:36:44.347 INFO shutdown-hook-0 org.apache.kyuubi.engine.KyuubiApplicationManager: Service[KyuubiApplicationManager] is stopped.
2024-11-06 18:36:44.347 INFO shutdown-hook-0 org.apache.kyuubi.session.KyuubiSessionManager: Service[KyuubiSessionManager] is stopped.
2024-11-06 18:36:44.352 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service[KyuubiBackendService] is stopped.
2024-11-06 18:36:44.352 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [MetricsSystem] is stopping.
2024-11-06 18:36:44.353 INFO shutdown-hook-0 org.apache.kyuubi.metrics.MetricsSystem: Service: [PrometheusReporterService] is stopping.
2024-11-06 18:36:44.353 INFO shutdown-hook-0 org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@54d18072{HTTP/1.1, (http/1.1)}{0.0.0.0:10019}
2024-11-06 18:36:44.354 INFO shutdown-hook-0 org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@712cfb63{/,null,STOPPED}
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.metrics.PrometheusReporterService: Prometheus metrics HTTP server has stopped.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.metrics.PrometheusReporterService: Service[PrometheusReporterService] is stopped.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.metrics.MetricsSystem: Service[MetricsSystem] is stopped.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [PeriodicGCService] is stopping.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.server.PeriodicGCService: Service[PeriodicGCService] is stopped.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service: [KinitAuxiliaryService] is stopping.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.server.KinitAuxiliaryService: Service[KinitAuxiliaryService] is stopped.
2024-11-06 18:36:44.355 INFO shutdown-hook-0 org.apache.kyuubi.server.KyuubiServer: Service[KyuubiServer] is stopped.
2024-11-06 18:36:44.355 INFO main-EventThread org.apache.kyuubi.shaded.zookeeper.ClientCnxn: EventThread shut down for session: 0x300231d79b1009c
```
### Kyuubi Engine Log Output
```logtalk
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:174)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.kyuubi.KyuubiException: Failed to initialize SparkSQLEngine: org.apache.kyuubi.util.JavaUtils.findLocalInetAddress()Ljava/net/InetAddress;
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:357)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:342)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:342)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:176)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
... 6 more
Caused by: java.lang.NoSuchMethodError: org.apache.kyuubi.util.JavaUtils.findLocalInetAddress()Ljava/net/InetAddress;
at org.apache.kyuubi.service.TFrontendService.$anonfun$serverAddr$2(TFrontendService.scala:56)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.kyuubi.service.TFrontendService.serverAddr$lzycompute(TFrontendService.scala:56)
at org.apache.kyuubi.service.TFrontendService.serverAddr(TFrontendService.scala:55)
at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:121)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:48)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.initialize(SparkSQLEngine.scala:70)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:353)
... 22 more
```
### Kyuubi Server Configurations
```yaml
kyuubi.authentication NONE
kyuubi.frontend.bind.host SPARKYARN-001
kyuubi.frontend.protocols THRIFT_BINARY,REST
kyuubi.frontend.thrift.binary.bind.port 10009
kyuubi.frontend.rest.bind.port 10099
kyuubi.operation.result.format arrow
kyuubi.engine.type SPARK_SQL
kyuubi.engine.share.level USER
kyuubi.session.engine.idle.timeout PT0M
kyuubi.engine.ui.stop.enabled true
kyuubi.session.engine.spark.showProgress true
kyuubi.ha.addresses ZOOKEEPER-001:2181,ZOOKEEPER-002:2181,ZOOKEEPER-003:2181
kyuubi.ha.namespace kyuubi
# kyuubi-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export SPARK_HOME=/usr/local/spark
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export KYUUBI_HOME=/opt/apache-kyuubi-1.10.0-bin
export KYUUBI_ENGINE_MEMORY=4g
export KYUUBI_JAVA_OPTS="-Xmx14g -Xms2g -XX:+UnlockDiagnosticVMOptions -XX:ParGCCardsPerStrideChunk=4096 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseCondCardMark -XX:MaxDirectMemorySize=4084m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=./logs -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -Xloggc:./logs/kyuubi-server-gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=5M -XX:NewRatio=3 -XX:MetaspaceSize=1024m"
```
### Kyuubi Engine Configurations
```yaml
spark.master yarn
spark.executor.cores 3
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.driver.memory 3g
spark.executor.memory 10g
spark.yarn.am.memory 1g
spark.eventLog.enabled true
spark.eventLog.dir hdfs:///user/spark/history/done
spark.history.fs.logDirectory hdfs:///user/spark/history/done
spark.worker.cleanup.enabled true
spark.worker.cleanup.interval 86400
spark.worker.cleanup.appDataTtl 604800
#spark.dynamicAllocation.enabled false
spark.yarn.queue q1
spark.yarn.jars file:/usr/local/spark/jars/*
#spark.history.fs.logDirectory file:/usr/local/spark/logs/spark-events
spark.history.fs.logDirectory hdfs:///user/spark/history/done
spark.eventLog.compress true
spark.driver.extraClassPath=$SPARK_HOME/jars/spark-network-yarn_2.12-3.5.1.jar
spark.submit.proxyUser.allowCustomClasspathInClusterMode true
spark.shuffle.compress true
spark.shuffle.spill.compress true
spark.sql.shuffle.partitions 240
spark.rdd.compress true
```
### Additional context
Here’s a detailed issue description based on the provided information:
---
### **Title**
Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to `KyuubiException` in `JavaUtils.findLocalInetAddress()`
### **Description**
When attempting to start the Kyuubi Engine using the `kyuubi-hive-jdbc-shaded-1.10.0` driver to establish a JDBC Connection, the following error occurs:
The Kyuubi Engine initiates a Spark application on YARN, but it fails shortly after with the following log output:
```
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1894)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:174)
...
Caused by: org.apache.kyuubi.KyuubiException: Failed to initialize SparkSQLEngine: org.apache.kyuubi.util.JavaUtils.findLocalInetAddress()Ljava/net/InetAddress;
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:357)
...
Caused by: java.lang.NoSuchMethodError: org.apache.kyuubi.util.JavaUtils.findLocalInetAddress()Ljava/net/InetAddress;
at org.apache.kyuubi.service.TFrontendService.$anonfun$serverAddr$2(TFrontendService.scala:56)
...
```
### **Steps to Reproduce**
1. Start Kyuubi Engine using `kyuubi-hive-jdbc-shaded-1.10.0` driver to establish a JDBC Connection.
2. Observe that a Spark Application is created in YARN.
3. The process fails, generating the above log output and terminating with an error.
### **Observed Behavior**
- The engine fails to initialize `SparkSQLEngine`, throwing a `KyuubiException` caused by a `NoSuchMethodError` in `org.apache.kyuubi.util.JavaUtils.findLocalInetAddress()`.
### **Expected Behavior**
- Kyuubi Engine should initialize successfully and handle JDBC connections without termination.
### **Environment**
- **Driver**: `kyuubi-hive-jdbc-shaded-1.10.0`
- **Platform**: YARN, Spark
### **Additional Information**
This issue might be related to missing or incompatible dependencies in `org.apache.kyuubi.util`. Any guidance on resolving this error or additional dependencies would be appreciated.
---
This template should help provide a clear and structured description for others to understand and investigate the issue.
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [X] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress() [kyuubi]
Posted by "inkkim (via GitHub)" <gi...@apache.org>.
inkkim commented on issue #6799:
URL: https://github.com/apache/kyuubi/issues/6799#issuecomment-2492637411
@pan3793
Thanks for the clarification! I followed your advice and made sure there were no incompatible Kyuubi libraries, and it worked perfectly without any errors. Specifically, I found kyuubi-spark-connector-tpcds_2.12-1.9.0.jar under /usr/local/spark/jars. Once I removed it and re-ran the setup, everything worked as expected. I really appreciate your guidance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress() [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #6799:
URL: https://github.com/apache/kyuubi/issues/6799#issuecomment-2463642999
Are you using a self built distribution package?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress() [kyuubi]
Posted by "inkkim (via GitHub)" <gi...@apache.org>.
inkkim commented on issue #6799:
URL: https://github.com/apache/kyuubi/issues/6799#issuecomment-2463727121
@wForget
Thank you for your response! No, I’m not using a self-built distribution package. I downloaded the Binary directly from the official Kyuubi releases page ([Kyuubi Releases](https://kyuubi.apache.org/releases.html)) and followed the standard installation steps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress() [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6799:
URL: https://github.com/apache/kyuubi/issues/6799#issuecomment-2463766576
have you put **any** jars under `/usr/local/spark/jars` that are not shipped by the official spark tgz?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress() [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #6799:
URL: https://github.com/apache/kyuubi/issues/6799#issuecomment-2469886658
you should ensure there are no other kyuubi libs that version lower the engine jar.
```
kyuubi-spark-sql-engine_2.12-1.10.0.jar
```
for example, you should NOT suppose it work if you put a `kyuubi-hive-jdbc-shaded-1.9.3.jar` under the `$SPARK_HOME/jars`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress() [kyuubi]
Posted by "inkkim (via GitHub)" <gi...@apache.org>.
inkkim commented on issue #6799:
URL: https://github.com/apache/kyuubi/issues/6799#issuecomment-2469866750
@pan3793 Yes, there are few things. Do I have to remove those before start kyuubi?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress() [kyuubi]
Posted by "inkkim (via GitHub)" <gi...@apache.org>.
inkkim closed issue #6799: [Bug] Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress()
URL: https://github.com/apache/kyuubi/issues/6799
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Error Running Kyuubi Engine: Failed to Initialize SparkSQLEngine due to KyuubiException in JavaUtils.findLocalInetAddress() [kyuubi]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6799:
URL: https://github.com/apache/kyuubi/issues/6799#issuecomment-2463564821
Hello @inkkim,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org