You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/07/13 00:49:10 UTC

[GitHub] [dolphinscheduler] fhygh opened a new issue, #10918: [Bug] [DataSource] MySQL Connection number increasing

fhygh opened a new issue, #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### What happened
   
   in dolphin worker-server, mysql connection number increasing always, which lead mysql server cannot work.
   ![image](https://user-images.githubusercontent.com/25889738/178624684-ca6fa8eb-73e5-4dde-b0f2-d45a73208ca5.png)
   
   
   ### What you expected to happen
   
   mysql connection number keep in normal number, 
   
   ### How to reproduce
   
   use MySQL DataSource to perform frequent operations, such as query and create table/delete table/update table.
   
   ### Anything else
   
   _No response_
   
   ### Version
   
   3.0.0-beta-2
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1182700384

   I just see the latest code, we didn't use connection pool in SQL task, so the hikari pool config isn't used for SQL task.
   
   Right now, for each task we will create a new connection and will close this connection after finished. I am not clear if we need to use connection pool in SQL task.
   
   cc @caishunfeng @SbloodyS 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] hwb1992 commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
hwb1992 commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1193132824

   > I submit #11120 to fix this.
   
   Nice try bro


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] caishunfeng commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
caishunfeng commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1182926562

   > I just see the latest code, we didn't use connection pool in SQL task, so the hikari pool config isn't used for SQL task.
   > 
   > Right now, for each task we will create a new connection and will close this connection after finished. I am not clear if we need to use connection pool in SQL task.
   > 
   > cc @caishunfeng @SbloodyS
   
   Another point is that ds has many datasource type, I'm not sure we can use a connection pool to support all the datasource type.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1193066259

   > This is a bug.When guava cache remove datasource,it needs to invoke close functin,which is never call in current version. I use 3.0.0-beta2-release.
   > I add a removeListiner in guava cache to invoke close function, and in CommonDataSourceClient class, i call dataSource.close function in close method to release db pool. Now the connection will release..
   > I hope community can fix this problem.
   
   Good catch, are you willing to submit a PR to fix?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1193089283

   I submit #11120 to fix this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun closed issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
ruanwenjun closed issue #10918: [Bug] [DataSource] MySQL Connection number increasing 
URL: https://github.com/apache/dolphinscheduler/issues/10918


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] fhygh commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
fhygh commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1184101633

   @SbloodyS  in dolphin 1.3.5, no this problem.
   ![image](https://user-images.githubusercontent.com/25889738/178927567-5dee3433-0494-4fd2-99e2-fe70be61690a.png)
   
   all mysql connection is created by DriverManager and closed in finally


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1182641261

   Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
   * In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
   * If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] fhygh commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
fhygh commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1193454442

   @ruanwenjun  great job, thank you .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1204662374

   > @ruanwenjun great job, thank you .
   
   You are welcome.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] SbloodyS commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1182704512

   > I just see the latest code, we didn't use connection pool in SQL task, so the hikari pool config isn't used for SQL task.
   > 
   > Right now, for each task we will create a new connection and will close this connection after finished. I am not clear if we need to use connection pool in SQL task.
   > 
   > cc @caishunfeng @SbloodyS
   
   I think we can provide an configure option to users in the data source center.
   
   But I think using connection pooling in SQL task may cause task blocking since ds is a distributed big data scheduling system. It is common to execute hundreds of tasks in parallel. The maximum number of connection supported in the database of my current production environment is 10000.
   
   As a result, it's better to expand the database or decrease the SQL task's parallelism. @fhygh 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] fhygh commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
fhygh commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1184106308

   > I just see the latest code, we didn't use connection pool in SQL task, so the hikari pool config isn't used for SQL task.
   > 
   > Right now, for each task we will create a new connection and will close this connection after finished. I am not clear if we need to use connection pool in SQL task.
   > 
   > cc @caishunfeng @SbloodyS
   
   @ruanwenjun  hello, we create HikariDataSource pool indeed, and you can test on SQL task with mysql datasource connection, you will see log "Creating HikariDataSource pool for maxActive" in worker-server log which is present in JDBCDataSourceProvider.java file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] hwb1992 commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
hwb1992 commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1193052134

   This is a bug.When guava cache remove datasource,it needs to invoke close functin,which is never call in current version. I use 3.0.0-beta2-release.
   I add a removeListiner in guava cache to invoke close function, and in CommonDataSourceClient class, i add dataSource.close function to release db pool to fix this problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] ruanwenjun commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1193065086

   > > I just see the latest code, we didn't use connection pool in SQL task, so the hikari pool config isn't used for SQL task.
   > > Right now, for each task we will create a new connection and will close this connection after finished. I am not clear if we need to use connection pool in SQL task.
   > > cc @caishunfeng @SbloodyS
   > 
   > @ruanwenjun hello, we create HikariDataSource pool indeed, and you can test on SQL task with mysql datasource connection, you will see log "Creating HikariDataSource pool for maxActive" in worker-server log which is present in JDBCDataSourceProvider.java file.
   
   This is due to we rely on dao module, but the `HikariDataSource` will not work for SQL task.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] hwb1992 commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
hwb1992 commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1193132786

   > 
   Nice try bro
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [dolphinscheduler] hwb1992 commented on issue #10918: [Bug] [DataSource] MySQL Connection number increasing

Posted by GitBox <gi...@apache.org>.
hwb1992 commented on issue #10918:
URL: https://github.com/apache/dolphinscheduler/issues/10918#issuecomment-1193132740

   Nice try bro.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org