You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/09/09 09:44:46 UTC
[GitHub] [incubator-kyuubi] zhang1002 opened a new issue #1066: [Bug] when you set kyuubi.engine.event.json.log.path, the server will not create this dirPath
zhang1002 opened a new issue #1066:
URL: https://github.com/apache/incubator-kyuubi/issues/1066
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
When you open the configuration item of :kyuubi.engine.event.json.log.path, but you don't create this path, the server will stop and throw some exception.
### Kyuubi Server Log Output
```logtalk
2021-09-08 14:51:50.519 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x78fe21e68f0325
2021-09-08 14:51:50.524 WARN service.ThriftFrontendService: Error opening session:
org.apache.kyuubi.KyuubiSQLException: Error opening session SessionHandle [ce33ab32-7ded-409a-89a5-cdb0dafd4d40] for hive due to Exception in thread "main" java.lang.IllegalStateException: Illegal Service State: LATENT
at org.apache.kyuubi.service.ThriftFrontendService.connectionUrl(ThriftFrontendService.scala:110)
at org.apache.kyuubi.service.Serverable.connectionUrl(Serverable.scala:34)
at org.apache.kyuubi.engine.spark.events.EngineEvent$.apply(EngineEvent.scala:107)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.engineStatus$lzycompute(SparkSQLEngine.scala:41)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.engineStatus(SparkSQLEngine.scala:41)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$main$1(SparkSQLEngine.scala:152)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$main$1$adapted(SparkSQLEngine.scala:150)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:150)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
See more: /home/hadoop/kyuubi/apache-kyuubi-1.3.0-incubating-bin/work/hive/kyuubi-spark-sql-engine.log.14
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68)
at org.apache.kyuubi.session.KyuubiSessionManager.openSession(KyuubiSessionManager.scala:74)
at org.apache.kyuubi.service.AbstractBackendService.openSession(AbstractBackendService.scala:45)
at org.apache.kyuubi.service.ThriftFrontendService.getSessionHandle(ThriftFrontendService.scala:190)
at org.apache.kyuubi.service.ThriftFrontendService.OpenSession(ThriftFrontendService.scala:199)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1377)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1362)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.kyuubi.service.authentication.HadoopThriftAuthBridgeServer$TUGIAssumingProcessor.process(HadoopThriftAuthBridgeServer.scala:163)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kyuubi.KyuubiSQLException: Exception in thread "main" java.lang.IllegalStateException: Illegal Service State: LATENT
at org.apache.kyuubi.service.ThriftFrontendService.connectionUrl(ThriftFrontendService.scala:110)
at org.apache.kyuubi.service.Serverable.connectionUrl(Serverable.scala:34)
at org.apache.kyuubi.engine.spark.events.EngineEvent$.apply(EngineEvent.scala:107)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.engineStatus$lzycompute(SparkSQLEngine.scala:41)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.engineStatus(SparkSQLEngine.scala:41)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$main$1(SparkSQLEngine.scala:152)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$main$1$adapted(SparkSQLEngine.scala:150)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:150)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
See more: /home/hadoop/kyuubi/apache-kyuubi-1.3.0-incubating-bin/work/hive/kyuubi-spark-sql-engine.log.14
at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:68)
at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:128)
... 1 more
```
### Kyuubi Engine Log Output
```logtalk
21/09/08 14:04:42 INFO SparkUI: Stopped Spark web UI at http://xxxxx
21/09/08 14:04:42 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
21/09/08 14:04:42 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
21/09/08 14:04:42 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed (this is expected if the application is shutting down.)
21/09/08 14:04:42 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/09/08 14:04:42 INFO MemoryStore: MemoryStore cleared
21/09/08 14:04:42 INFO BlockManager: BlockManager stopped
21/09/08 14:04:42 INFO BlockManagerMaster: BlockManagerMaster stopped
21/09/08 14:04:42 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/09/08 14:04:42 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.lang.IllegalStateException: Illegal Service State: LATENT
at org.apache.kyuubi.service.ThriftFrontendService.connectionUrl(ThriftFrontendService.scala:110)
at org.apache.kyuubi.service.Serverable.connectionUrl(Serverable.scala:34)
at org.apache.kyuubi.engine.spark.events.EngineEvent$.apply(EngineEvent.scala:107)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.engineStatus$lzycompute(SparkSQLEngine.scala:41)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.engineStatus(SparkSQLEngine.scala:41)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$main$1(SparkSQLEngine.scala:152)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$main$1$adapted(SparkSQLEngine.scala:150)
at scala.Option.foreach(Option.scala:407)
at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:150)
at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
```
### Kyuubi Server Configurations
```yaml
kyuubi.engine.event.json.log.path=/home/hive/kyuubi_events
kyuubi.engine.event.loggers=JSON
```
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you closed issue #1066: [Bug] when you set kyuubi.engine.event.json.log.path, the server will not create this dirPath
Posted by GitBox <gi...@apache.org>.
ulysses-you closed issue #1066:
URL: https://github.com/apache/incubator-kyuubi/issues/1066
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1066: [Bug] when you set kyuubi.engine.event.json.log.path, the server will not create this dirPath
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1066:
URL: https://github.com/apache/incubator-kyuubi/issues/1066#issuecomment-915931556
nice catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org