You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/08/18 03:01:06 UTC
[GitHub] [dolphinscheduler] simon824 opened a new issue, #11533: [Bug] [worker] Connections is not released after completing the SQL task
simon824 opened a new issue, #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
SQL type task and the data source is hive jdbc, the connection seems is not released after completing the SQL task, cause the number of connections always increasing.
The ds-worker at this point has no SQL tasks in progress, but there are still more than fifty connections
### What you expected to happen
close the connection after completing the SQL task
### How to reproduce
1
### Anything else
_No response_
### Version
3.0.0
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] stalary commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
stalary commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1237761621
> > > > > Is this happen in 3.0.0-release or just beta?
> > > > > There is a same issue #11120, but has already been fixed at 3.0.0-release.
> > > >
> > > >
> > > > 3.0.0-release
> > >
> > >
> > > 3.0.0release use DataSource in sql task, it will hold connection at default, you can reduce the [`minimumIdle`](https://github.com/brettwooldridge/HikariCP) to make the connection to be expire.
> >
> >
> > I locally standalone changed minimumIdle to 1 for all modules, but the total link is still 5.
>
> Could you please try to set the config `spring.datasource.minIdle=1` in `common.properties`.
You are right, the connection decrease when I config common.properties. @ruanwenjun @chunhui5566 @mongooooo @simon824 But I think we should decrease default minIdle in code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] chunhui5566 commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
chunhui5566 commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1237712524
I meet the same problem in 3.0.0-release also, i had to restart it when the jdbc connection increased to more than 100 now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] chunhui5566 commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
chunhui5566 commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1237801161
> > > > > > Is this happen in 3.0.0-release or just beta?
> > > > > > There is a same issue #11120, but has already been fixed at 3.0.0-release.
> > > > >
> > > > >
> > > > > 3.0.0-release
> > > >
> > > >
> > > > 3.0.0release use DataSource in sql task, it will hold connection at default, you can reduce the [`minimumIdle`](https://github.com/brettwooldridge/HikariCP) to make the connection to be expire.
> > >
> > >
> > > I locally standalone changed minimumIdle to 1 for all modules, but the total link is still 5.
> >
> >
> > Could you please try to set the config `spring.datasource.minIdle=1` in `common.properties`.
>
> You are right, the connection decrease when I config common.properties. @ruanwenjun @chunhui5566 @mongooooo @simon824 But I think we should decrease default minIdle in code.
Thank you for you recheck and reply , i think the current problem is the connection not release, one day sooner or later it will still increased to more than 100 . Befor V3.0.0-beta1 the datasource be put in map and there is no such problem. now it in Guava Cache . Could it be the problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simon824 commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
simon824 commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1263004656
I had set the config `spring.datasource.minIdle=1` in `common.properties`, but after running for a few weeks I found that the problem still exists, the number of connections is still increasing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] mongooooo commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
mongooooo commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1232365242
+1 , I met the same problem, it is very destructive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1263045210
@simon824 @stalary Thanks for report this bug, I have submitted #12226 to fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1238215891
Since this has already been fixed at 3.0.0 release, I will close this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun closed issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
ruanwenjun closed issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
URL: https://github.com/apache/dolphinscheduler/issues/11533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1263022185
> I had set the config `spring.datasource.minIdle=1` in `common.properties`, but after running for a few weeks I found that the problem still exists, the number of connections is still increasing.
Did you use 3.0.0-release or 3.0.0 beta?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] stalary commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
stalary commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1220144104
> Is this happen in 3.0.0-release or just beta?
>
> There is a same issue #11120, but has already been fixed at 3.0.0-release.
3.0.0-release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1218982380
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] stalary commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
stalary commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1219163691
I meet the same problem of maintaining a connection to the database even when there was no task scheduled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1220147160
> > Is this happen in 3.0.0-release or just beta?
> > There is a same issue #11120, but has already been fixed at 3.0.0-release.
>
> 3.0.0-release
3.0.0release use DataSource, and it will hold connection at default, you can reduce the [`minimumIdle`](https://github.com/brettwooldridge/HikariCP) to make the connection to be expire.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1220139505
Is this happen in 3.0.0-release or just beta?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simon824 commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
simon824 commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1220189840
> Could you please try to set the config `spring.datasource.minIdle=1` in `common.properties`.
@ruanwenjun What is the default value of `spring.datasource.minIdle`, I feel that there is no limit to the number of connections, and the jdbc connection on the worker side can easily grow to more than 100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simon824 commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
simon824 commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1240259096
hi @stalary , set `export SPRING_DATASOURCE_MIN_IDLE=1` in `$DS_HOME/bin/env/dolphinscheduler_env.sh` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] stalary commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
stalary commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1220157303
> > > Is this happen in 3.0.0-release or just beta?
> > > There is a same issue #11120, but has already been fixed at 3.0.0-release.
> >
> >
> > 3.0.0-release
>
> 3.0.0release use DataSource in sql task, it will hold connection at default, you can reduce the [`minimumIdle`](https://github.com/brettwooldridge/HikariCP) to make the connection to be expire.
I locally standalone changed minimumIdle to 1 for all modules, but the total link is still 5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] stalary commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
stalary commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1237825863
> Thank you for you recheck and reply , i think the current problem is the connection not release, one day sooner or later it will still increased to more than 100 . Befor V3.0.0-beta1 the datasource be put in map and there is no such problem. now it in Guava Cache . Could it be the problem?
Yes, in 3.0-release it no problem~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1237983253
> > > > > > > Is this happen in 3.0.0-release or just beta?
> > > > > > > There is a same issue #11120, but has already been fixed at 3.0.0-release.
> > > > > >
> > > > > >
> > > > > > 3.0.0-release
> > > > >
> > > > >
> > > > > 3.0.0release use DataSource in sql task, it will hold connection at default, you can reduce the [`minimumIdle`](https://github.com/brettwooldridge/HikariCP) to make the connection to be expire.
> > > >
> > > >
> > > > I locally standalone changed minimumIdle to 1 for all modules, but the total link is still 5.
> > >
> > >
> > > Could you please try to set the config `spring.datasource.minIdle=1` in `common.properties`.
> >
> >
> > You are right, the connection decrease when I config common.properties. @ruanwenjun @chunhui5566 @mongooooo @simon824 But I think we should decrease default minIdle in code.
>
> Thank you for you recheck and reply , i think the current problem is the connection not release, one day sooner or later it will still increased to more than 100 . Befor V3.0.0-beta1 the datasource be put in map and there is no such problem. now it in Guava Cache . Could it be the problem?
Yes, in 3.0.0 release, we have fixed this bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] stalary commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
stalary commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1237738765
> I meet the same problem in 3.0.0-release also, i had to restart it when the jdbc connection increased to more than 100 now.
ok, let me check the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simon824 commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
simon824 commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1263023687
> > I had set the config `spring.datasource.minIdle=1` in `common.properties`, but after running for a few weeks I found that the problem still exists, the number of connections is still increasing.
>
> You use 3.0.0-release or 3.0.0 beta?
3.0.0-release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] stalary commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
stalary commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1221989367
> Hi @stalary , how is this problem going, have you found the cause?
I haven't found the problem yet. I'll take a look at it when I'm free
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] simon824 commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
simon824 commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1221983807
Hi @stalary , how is this problem going, have you found the cause?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #11533: [Bug] [worker] Connections is not released after completing the SQL task
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #11533:
URL: https://github.com/apache/dolphinscheduler/issues/11533#issuecomment-1220161360
> > > > Is this happen in 3.0.0-release or just beta?
> > > > There is a same issue #11120, but has already been fixed at 3.0.0-release.
> > >
> > >
> > > 3.0.0-release
> >
> >
> > 3.0.0release use DataSource in sql task, it will hold connection at default, you can reduce the [`minimumIdle`](https://github.com/brettwooldridge/HikariCP) to make the connection to be expire.
>
> I locally standalone changed minimumIdle to 1 for all modules, but the total link is still 5.
Could you please try to set the config `spring.datasource.minIdle=1` in `common.properties`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org