You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "caoes (via GitHub)" <gi...@apache.org> on 2023/02/22 08:14:33 UTC
[GitHub] [dolphinscheduler] caoes opened a new issue, #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
caoes opened a new issue, #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
Here is a question. Is it because of hive or DS。
The following is the error message:
[INFO] 2022-09-08 04:25:44.024 org.apache.hive.jdbc.Utils:[318] - Supplied authorities: 100.16100.201:10001
[INFO] 2022-09-08 04:25:44.024 org.apache.hive.jdbc.Utils:[437] - Resolved authority: 100.16100.201:10001
[WARN] 2022-09-08 04:25:45.688 com.zaxxer.hikari.pool.PoolBase:[184] - HikariPool-2 - Failed to validate connection org.apache.hive.jdbc.HiveConnection@46f96733 (org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out). Possibly consider using a shorter maxLifetime value.
[ERROR] 2022-09-08 04:25:45.690 org.apache.dolphinscheduler.plugin.datasource.hive.HiveDataSourceClient:[152] - get oneSessionDataSource Connection fail SQLException: HikariPool-2 - Connection is not available, request timed out after 30002ms.
java.sql.SQLTransientConnectionException: HikariPool-2 - Connection is not available, request timed out after 30002ms.
at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128)
at org.apache.dolphinscheduler.plugin.datasource.hive.HiveDataSourceClient.getConnection(HiveDataSourceClient.java:150)
at org.apache.dolphinscheduler.plugin.datasource.api.plugin.DataSourceClientProvider.getConnection(DataSourceClientProvider.java:66)
at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:183)
at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:154)
at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:191)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:317)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250)
at com.zaxxer.hikari.pool.PoolBase.isConnectionAlive(PoolBase.java:169)
at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:186)
... 14 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
### What you expected to happen
When run workflow scheduling periodic, accidental problem, speculation HSQL complex, hive cluster resources mission is frequent, lead to submit a timeout, I try to modify the timed out parameters, increase the parameters of the spring. The datasource. `spring.datasource.connection.timeout=600000` and ` dataSource.setConnectionTimeout(PropertyUtils.getLong(Constants.SPRING_DATASOURCE_CONNECTION_TIMEOUT));
` The problem no longer arises, but new problems arise, and new problems are episodic problems
PS:hive connection resource leakage has been resolved. and SQLTask job support batch execute sql ，close(rs, ptmt,conn) modify colse(conn);
java.sql.SQLException: org.apache.thrift.transport.TTransportException: SASL authentication not complete
java.sql.SQLException: org.apache.thrift.transport.TTransportException: Cannot read from null inputStream
java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
java.sql.SQLException: org.apache.thrift.transport.TTransportException: Cannot write to null outputStream
java.sql.SQLException: org.apache.thrift.transport.TTransportException: java.net.SocketException: Socket closed
### How to reproduce
no
### Anything else
no
### Version
2.0.x
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caoes commented on issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
Posted by "caoes (via GitHub)" <gi...@apache.org>.
caoes commented on issue #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602#issuecomment-1441314791
> The issue happened in 3.1.1 as well
we are 2.0.5 ，not upgraded to version 3.x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [worker-server] Connection is not available, request timed out after 30002ms [dolphinscheduler]
Posted by "hardaway7melo (via GitHub)" <gi...@apache.org>.
hardaway7melo commented on issue #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602#issuecomment-2138471982
"spring.datasource.connection.timeout=600000"
Bro， where to adjust this parameter, can u please tell me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] EricGao888 commented on issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
Posted by "EricGao888 (via GitHub)" <gi...@apache.org>.
EricGao888 commented on issue #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602#issuecomment-1495466994
@caoes Hello, may I ask why you closed this issue? Does it mean that you figured out a solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] k9flame commented on issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
Posted by "k9flame (via GitHub)" <gi...@apache.org>.
k9flame commented on issue #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602#issuecomment-1480585065
I have the same problem，is there a way to solve？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Cocytus316 commented on issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
Posted by "Cocytus316 (via GitHub)" <gi...@apache.org>.
Cocytus316 commented on issue #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602#issuecomment-1535644634
Ditto, I'm a 3.0.1 version and I get this problem from time to time.
And I suspect it might have something to do with the stress on the server at that point in time, but I don't know for sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602#issuecomment-1441260405
Please check hiveserver2 service availablity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caoes commented on issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
Posted by "caoes (via GitHub)" <gi...@apache.org>.
caoes commented on issue #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602#issuecomment-1441307496
> Please check hiveserver2 service availablity.
I think this has something to do with it but next sql job execute successed，our big data engineer checked hive service and found no problem。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Flutemr commented on issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
Posted by "Flutemr (via GitHub)" <gi...@apache.org>.
Flutemr commented on issue #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602#issuecomment-1463314155
> > Please check hiveserver2 service availablity.
> > After an exception occurs, check hive service logs in the Hadoop cluster. The following exception is displayed in the logs, but it is not certain that the exception is the current task. Later, the task is queried on the yarn application based on the failed task. The task fails, but some tasks on yarn are finished and others are killed. killed cause: Another service was killed by hiveserver2。
>
> `org.apache.hadoop.hive.ql.exec.Task: [HiveServer2-Background-Pool: Thread-1362777]: Execution completed successfully 2023-02-24 05:12:57,454 INFO org.apache.hadoop.hive.ql.exec.Task: [HiveServer2-Background-Pool: Thread-1362777]: MapredLocal task succeeded 2023-02-24 05:12:57,453 WARN org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask: [HiveServer2-Background-Pool: Thread-1362777-LocalTask-MAPREDLOCAL-stderr]: I/O error in redirector thread. java.io.IOException: Stream closed at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) ~[?:1.8.0_181] at java.io.BufferedInputStream.read(BufferedInputStream.java:336) ~[?:1.8.0_181] at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[?:1.8.0_181] at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[?:1.8.0_181] at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:1.8.0_181] at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[?:1.8.0_181] at java.io.BufferedReader.fill(BufferedR
eader.java:161) ~[?:1.8.0_181] at java.io.BufferedReader.readLine(BufferedReader.java:324) ~[?:1.8.0_181] at java.io.BufferedReader.readLine(BufferedReader.java:389) ~[?:1.8.0_181] at org.apache.hadoop.hive.common.log.LogRedirector.run(LogRedirector.java:64) [hive-common-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]`
Hey, buddy, have you solved that problem? We had that problem, too！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602#issuecomment-1439593685
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caoes commented on issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
Posted by "caoes (via GitHub)" <gi...@apache.org>.
caoes commented on issue #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602#issuecomment-1445637723
> Please check hiveserver2 service availablity.
After an exception occurs, check hive service logs in the Hadoop cluster. The following exception is displayed in the logs, but it is not certain that the exception is the current task. Later, the task is queried on the yarn application based on the failed task. The task fails, but some tasks on yarn are finished and others are killed. killed cause: Another service was killed by hiveserver2。
`org.apache.hadoop.hive.ql.exec.Task: [HiveServer2-Background-Pool: Thread-1362777]: Execution completed successfully
2023-02-24 05:12:57,454 INFO org.apache.hadoop.hive.ql.exec.Task: [HiveServer2-Background-Pool: Thread-1362777]: MapredLocal task succeeded
2023-02-24 05:12:57,453 WARN org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask: [HiveServer2-Background-Pool: Thread-1362777-LocalTask-MAPREDLOCAL-stderr]: I/O error in redirector thread.
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) ~[?:1.8.0_181]
at java.io.BufferedInputStream.read(BufferedInputStream.java:336) ~[?:1.8.0_181]
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[?:1.8.0_181]
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[?:1.8.0_181]
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:1.8.0_181]
at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[?:1.8.0_181]
at java.io.BufferedReader.fill(BufferedReader.java:161) ~[?:1.8.0_181]
at java.io.BufferedReader.readLine(BufferedReader.java:324) ~[?:1.8.0_181]
at java.io.BufferedReader.readLine(BufferedReader.java:389) ~[?:1.8.0_181]
at org.apache.hadoop.hive.common.log.LogRedirector.run(LogRedirector.java:64) [hive-common-2.1.1-cdh6.3.2.jar:2.1.1-cdh6.3.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhaohehuhu commented on issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on issue #13602:
URL: https://github.com/apache/dolphinscheduler/issues/13602#issuecomment-1441256429
The issue happened in 3.1.1 as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caoes closed issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
Posted by "caoes (via GitHub)" <gi...@apache.org>.
caoes closed issue #13602: [Bug] [worker-server] Connection is not available, request timed out after 30002ms
URL: https://github.com/apache/dolphinscheduler/issues/13602
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org