Re: [I] [Bug] Spark connection level engines didn't be closed after all job finished [kyuubi]

["pravin1406 (via GitHub)" <gi...@apache.org>]

pravin1406 commented on issue #4591:
URL: https://github.com/apache/kyuubi/issues/4591#issuecomment-2115683310

   @lordk911 
   @yaooqinn 
   
   
   We are also getting the exact issue in our prod deployment. We also see this happening in 1 of the kyuubi server only out of 4. On checking the report.json metric file inside that server we get_operation_status having mean of 2600 millisecond.  have attached metric report for this.  
   
   [report-4.json](https://github.com/apache/kyuubi/files/15338212/report-4.json)
   basically this operation was taking loads of time and seems to get stuck a lot. This could be something related to high user load i guess. We have seen this a few times now. We investigate , but find only restarting as the solution mostly. 
   
   
   I was thinking of decreasing the kyuubi.operation.idle.timeout to a very small value maybe few minute. Will that have any wrong effects on my queries?  Some of our queries to run for hours.
   
   
   
   
   
   Below is the kyuubi server stacktrace which we see has high cpu time and elapsed time. I could not co relate this guid with a string operation handle id, as i did not know how to easily convert and seemed like to much effort.
   `
   "async-request-executor-TSessionHandle(sessionId:THandleIdentifier(guid:66 CD F8 DE F2 CC 44 72 86 96 8F E7 97 DC EB 00, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38)): Thread-54226" #54226 daemon prio=5 os_prio=0 cpu=290468489.96ms elapsed=707467.31s tid=0x00007f3fe8018000 nid=0x7c807 runnable  [0x00007f422b7fa000]
      java.lang.Thread.State: RUNNABLE
   	at java.net.SocketInputStream.socketRead0(java.base@11.0.16/Native Method)
   	at java.net.SocketInputStream.socketRead(java.base@11.0.16/Unknown Source)
   	at java.net.SocketInputStream.read(java.base@11.0.16/Unknown Source)
   	at java.net.SocketInputStream.read(java.base@11.0.16/Unknown Source)
   	at java.io.BufferedInputStream.fill(java.base@11.0.16/Unknown Source)
   	at java.io.BufferedInputStream.read1(java.base@11.0.16/Unknown Source)
   	at java.io.BufferedInputStream.read(java.base@11.0.16/Unknown Source)
   	- locked <0x0000000083be8ad0> (a java.io.BufferedInputStream)
   	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
   	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
   	at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
   	at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
   	at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
   	at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
   	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
   	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
   	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
   	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
   	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
   	at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_GetOperationStatus(TCLIService.java:475)
   	at org.apache.hive.service.rpc.thrift.TCLIService$Client.GetOperationStatus(TCLIService.java:462)
   	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$getOperationStatus$1(KyuubiSyncThriftClient.scala:369)
   	at org.apache.kyuubi.client.KyuubiSyncThriftClient$$Lambda$943/0x0000000840a79040.apply(Unknown Source)
   	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquiredAsyncRequest$2(KyuubiSyncThriftClient.scala:142)
   	at org.apache.kyuubi.client.KyuubiSyncThriftClient$$Lambda$938/0x0000000840a6f040.call(Unknown Source)
   	at java.util.concurrent.FutureTask.run(java.base@11.0.16/Unknown Source)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.16/Unknown Source)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.16/Unknown Source)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.16/Unknown Source)
   	at java.lang.Thread.run(java.base@11.0.16/Unknown Source)`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org