You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/07/21 08:44:32 UTC
[GitHub] [dolphinscheduler] simon824 opened a new issue, #11093: [Bug] [worker] Too many hiveserver2 connctions
simon824 opened a new issue, #11093:
URL: https://github.com/apache/dolphinscheduler/issues/11093
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
There are a large number of such logs in my worker logs, generated every few seconds, which eventually leads to too many connections to hiveserver2. And i have no idea what happen.
```
[INFO] 2022-07-21 09:58:15.435 +0800 org.apache.hive.jdbc.Utils:[285] - Supplied authorities: hive.url:20000
[INFO] 2022-07-21 09:58:15.435 +0800 org.apache.hive.jdbc.Utils:[372] - Resolved authority: hive.url:20000
[INFO] 2022-07-21 09:58:15.435 +0800 org.apache.hive.jdbc.HiveConnection:[189] - Will try to open client transport with JDBC Uri: jdbc:hive2://hive.url:20000/default
[INFO] 2022-07-21 09:58:19.262 +0800 org.apache.hive.jdbc.Utils:[285] - Supplied authorities: hive.url:20000
[INFO] 2022-07-21 09:58:19.262 +0800 org.apache.hive.jdbc.Utils:[372] - Resolved authority: hive.url:20000
[INFO] 2022-07-21 09:58:19.262 +0800 org.apache.hive.jdbc.HiveConnection:[189] - Will try to open client transport with JDBC Uri: jdbc:hive2://hive.url:20000/default
........
```
### What you expected to happen
.
### How to reproduce
.
### Anything else
_No response_
### Version
3.0.0-beta-2
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simon824 commented on issue #11093: [Question] [Worker] Too many hiveserver2 connctions
Posted by "simon824 (via GitHub)" <gi...@apache.org>.
simon824 commented on issue #11093:
URL: https://github.com/apache/dolphinscheduler/issues/11093#issuecomment-1506174193
refer to #11533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] TangTangGrit commented on issue #11093: [Question] [Worker] Too many hiveserver2 connctions
Posted by GitBox <gi...@apache.org>.
TangTangGrit commented on issue #11093:
URL: https://github.com/apache/dolphinscheduler/issues/11093#issuecomment-1193448343
Which database does hive metadata exist in
> 当前最大并发连接数等于工作线程数。建议改进 hiveserver2 的配置或降低并发。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ka1ka commented on issue #11093: [Question] [Worker] Too many hiveserver2 connctions
Posted by GitBox <gi...@apache.org>.
ka1ka commented on issue #11093:
URL: https://github.com/apache/dolphinscheduler/issues/11093#issuecomment-1199073066
It seems that configuring this parameter 'support.hive.oneSession=ture' is effective. If the worker is a cluster, each worker will create a container for hive operations. Of course, if it has been created and it is easy to not expire, it can continue to run in the current container, no new containers will be created
`sed -i 's/support.hive.oneSession=false/support.hive.oneSession=true/g' $DS_HOME/worker-server/conf/common.properties`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] conorzhao commented on issue #11093: [Question] [Worker] Too many hiveserver2 connctions
Posted by "conorzhao (via GitHub)" <gi...@apache.org>.
conorzhao commented on issue #11093:
URL: https://github.com/apache/dolphinscheduler/issues/11093#issuecomment-1500790201
I also encountered a problem with the number of connections, and after changing the configuration according to the answer, it did not work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11093: [Question] [Worker] Too many hiveserver2 connctions
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11093:
URL: https://github.com/apache/dolphinscheduler/issues/11093#issuecomment-1191258933
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11093: [Question] [Worker] Too many hiveserver2 connctions
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11093:
URL: https://github.com/apache/dolphinscheduler/issues/11093#issuecomment-1229597232
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] forest455 commented on issue #11093: [Question] [Worker] Too many hiveserver2 connctions
Posted by GitBox <gi...@apache.org>.
forest455 commented on issue #11093:
URL: https://github.com/apache/dolphinscheduler/issues/11093#issuecomment-1260788555
I found same problem. I found no workflow running, but why worker keep printing hive connecting logs.
![Uploading IMG_20220928_194714_edit_273430840399422.jpg…]()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #11093: [Bug] [worker] Too many hiveserver2 connctions
Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #11093:
URL: https://github.com/apache/dolphinscheduler/issues/11093#issuecomment-1191234576
The current maximum number of concurrent connections is equal to the number of worker threads. It's recommended to improve the configuration of hiveserver2 or reduce the concurrency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [Worker] Too many hiveserver2 connctions [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11093:
URL: https://github.com/apache/dolphinscheduler/issues/11093#issuecomment-1789866467
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [Worker] Too many hiveserver2 connctions [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11093:
URL: https://github.com/apache/dolphinscheduler/issues/11093#issuecomment-1802966562
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [Worker] Too many hiveserver2 connctions [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #11093: [Question] [Worker] Too many hiveserver2 connctions
URL: https://github.com/apache/dolphinscheduler/issues/11093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org