You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "bowenliang123 (via GitHub)" <gi...@apache.org> on 2023/10/10 08:15:52 UTC
[I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
bowenliang123 opened a new issue, #5392:
URL: https://github.com/apache/kyuubi/issues/5392
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the bug
With the follow timeout set, the statement gets block and no proper response or connection closing.
Connection and socket timeouts are set via JDBC URL:
`jdbc:kyuubi://kyuubi_server_ip:port/;retries=3;connectTimeout=300000;socketTimeout=1800000`
Query Timeout is set via JDBC statement `statement.setTimeout()`.
Connection Timeout: 30000 ms (or 30s)
Socket Timeout: 1800000ms (or 30min)
Query Timeout: 60*60 sec (or 1hr)
After waiting for ~ 3hr, the statement finally gets TIMEDOUT_STATE and closes the session.
### Affects Version(s)
1.7.1 / master
### Kyuubi Server Log Output
```logtalk
The insert statement was submitted at `06:22:24`.
At first, the JDBC statement logs output correctly with repeated `RUNNING_STATE`.
At `06:29:24`, the output get blocked, until 2 hours later at `08:35:41`.
Finally, at `09:16:03`, with SQLTimeoutException thrown and failed the query.
[2023-10-10 06:22:24]-[INFO] [pool-140421-thread-1] 2023-10-10 06:22:22.032 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[b0bf024e-c21e-48b8-8ac7-1702f731ba96] in RUNNING_STATE
...
[2023-10-10 06:28:34]-[INFO] [pool-140421-thread-1] 2023-10-10 06:28:33.596 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[b0bf024e-c21e-48b8-8ac7-1702f731ba96] in RUNNING_STATE
[2023-10-10 06:28:49]-[INFO] [pool-140421-thread-1] 2023-10-10 06:28:46.013 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[b0bf024e-c21e-48b8-8ac7-1702f731ba96] in RUNNING_STATE
[2023-10-10 06:29:19]-[INFO] [pool-140421-thread-1] 2023-10-10 06:29:15.079 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[b0bf024e-c21e-48b8-8ac7-1702f731ba96] in RUNNING_STATE
[2023-10-10 06:29:24]-[INFO] [pool-140421-thread-1] 2023-10-10 06:29:20.728 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[b0bf024e-c21e-48b8-8ac7-1702f731ba96] in RUNNING_STATE
[2023-10-10 08:35:41]-[INFO] [pool-140421-thread-1] 2023-10-10 08:35:39.212 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[b0bf024e-c21e-48b8-8ac7-1702f731ba96] in RUNNING_STATE
[2023-10-10 09:16:03]-[INFO] [pool-140421-thread-1] 2023-10-10 09:16:03.827 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[b0bf024e-c21e-48b8-8ac7-1702f731ba96] in TIMEDOUT_STATE
[2023-10-10 09:16:03]-[INFO] [pool-140421-thread-1] 2023-10-10 09:16:03.852 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing gf_personas_plf's query[b0bf024e-c21e-48b8-8ac7-1702f731ba96]: RUNNING_STATE -> TIMEDOUT_STATE, time taken: 10429.681 seconds
[2023-10-10 09:16:03]-[FATAL] java.sql.SQLTimeoutException: Query timed out after 3600 seconds
at org.apache.kyuubi.jdbc.hive.KyuubiStatement.waitForOperationToComplete(KyuubiStatement.java:411)
at org.apache.kyuubi.jdbc.hive.KyuubiStatement.executeWithConfOverlay(KyuubiStatement.java:200)
at org.apache.kyuubi.jdbc.hive.KyuubiStatement.execute(KyuubiStatement.java:194)
at org.apache.kyuubi.jdbc.hive.KyuubiPreparedStatement.execute(KyuubiPreparedStatement.java:66)
at cn.com.gf.bdp.horae.runner.engine.indexcalculate.connector.KyuubiConnector.executeSql(KyuubiConnector.java:157)
at cn.com.gf.bdp.horae.runner.engine.indexcalculate.connector.KyuubiConnector.executeSql(KyuubiConnector.java:97)
at cn.com.gf.bdp.horae.runner.engine.indexcalculate.KyuubiIndexCalculateEngine.execute(KyuubiIndexCalculateEngine.java:188)
at cn.com.gf.bdp.horae.runner.SparkIndexTaskRunner.execute(SparkIndexTaskRunner.java:48)
at cn.com.gf.bdp.horae.runner.AbstractTaskRunner.run(AbstractTaskRunner.java:215)
at java.lang.Thread.run(Thread.java:748)
```
```
### Kyuubi Engine Log Output
_No response_
### Kyuubi Server Configurations
_No response_
### Kyuubi Engine Configurations
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
- [X] No. I cannot submit a PR at this time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1756651296
@bowenliang123 there is a chance that the fetch result may block for a long time, for example, when incremental collect is enabled, once the first partition of the result is available, the operation state will be marked as FINISHED, the subsequential partitions will be calculated lazily, which may cause the fetch result blocks for a long time.
And, query timeout should be a concept of operation scope, but the thrift client is session scope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754941610
> timeout 0 with additional probe is by design, the thrift client lifecycle does not match the statement
Socket timeout is about maximum time to wait for the response only. A limited value is more close to the guarantee.
And we could also rebuild the client for the session if it's failed, as it's a mutable member of the session.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754879149
I see not timeout limits on `fetchQueryLog` . hmm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754681773
It's suspicious that the loop inside `ExecuteStatement` for status checks,
1. it tries to fetch query log first (`fetchQueryLog`)
2. and then poll the operation state for query
In step one, it uses a `KyuubiSyncThriftClient` instance to connect and query from the engine instance.
The a `KyuubiSyncThriftClient` instance is created with no socket timeout (set to 0 manually),
KyuubiSessionImpl:
```
_client = KyuubiSyncThriftClient.createClient(user, passwd, host, port, sessionConf)
```
KyuubiSyncThriftClient:
```
val tProtocol = createTProtocol(user, passwd, host, port, 0, loginTimeout)
```
With no socket timeout set, it may block the timeout state checks and therefore the statement is not able to do the step 2 and jammed at step 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 closed issue #5392: [Bug] Statements get blocked even with timeouts of connection, socket and query already set
URL: https://github.com/apache/kyuubi/issues/5392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754751083
The engine probe is a side-way approach for engine lifecycle.
May I create a PR to change the socket timeout from 0 to defined value falling back to query timeout ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754849647
> timeout 0 with additional probe is by design, the thrift client lifecycle does not match the statement
OK. Is there anything we could do on the server side for a timeout guarantee?
e.g. 1. set to time out state if statement duration exceeded query timeout on server side.
2. rebuild the client instance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1755371394
> I think we can also start queryTimeout monitor in the server.
SGTM. Once the elapsed duration of the statement is greater tahn the query timeout , the statement should be marked as TIMED_OUT status.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754877081
Doubtfully have TIMEDOUT_STATE finally, the driver could be busy GC and not responding correctly. And maybe when it is back to normal, it sends the response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754913692
> Doubtfully have TIMEDOUT_STATE finally, the driver could be busy GC and not responding correctly. And maybe when it is back to normal, it sends the response.
Full GC that lasts an hour seems impossible, if that happens the yarn will kill engine too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754836088
> Can you provide engine logs? Query timeout is triggered by the engine side.
Sorry, the engine log is not available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754919929
I think we can also start queryTimeout monitor in the server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1768501294
> I think we can also start queryTimeout monitor in the server.
The server-side timeout monitor has been implemented with https://github.com/apache/kyuubi/pull/5398 now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "bowenliang123 (via GitHub)" <gi...@apache.org>.
bowenliang123 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754687386
cc @turboFei @pan3793 @wForget
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754730846
Try setting `kyuubi.session.engine.alive.probe.enabled=true`, BTW, it was introduced in 1.6.0 but disabled by default, is it good to enable it in 1.8? @turboFei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754755243
timeout 0 with additional probe is by design, the thrift client lifecycle does not match the statement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754799016
Can you provide engine logs? Query timeout is triggered by the engine side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [I] [Bug] Statements get blocked even with timeouts of connection, socket and query already set [kyuubi]
Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on issue #5392:
URL: https://github.com/apache/kyuubi/issues/5392#issuecomment-1754873151
> With no socket timeout set, it may block the timeout state checks and therefore the statement is not able to do the step 2 and jammed at step 1.
I think the case may not be due to this. If client socket is blocked, the operation should not receive TIMEDOUT_STATE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org