You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2021/11/17 06:36:34 UTC
[GitHub] [dolphinscheduler] dailidong opened a new issue #6881: Change DataSource Connection Pool from Druid to HiKariCP
dailidong opened a new issue #6881:
URL: https://github.com/apache/dolphinscheduler/issues/6881
### Discussed in https://github.com/apache/dolphinscheduler/discussions/6448
<div type='discussions-op-text'>
<sup>Originally posted by **dailidong** October 5, 2021</sup>
As I found that the performance of HiKariCP Connection Pool is better than Druid , I did a small test between these two components(Informal testing), and my test program are as follows:
```
import org.apache.dolphinscheduler.common.enums.CommandType;
import org.apache.dolphinscheduler.common.thread.ThreadUtils;
import org.apache.dolphinscheduler.dao.datasource.ConnectionFactory;
import org.apache.dolphinscheduler.dao.entity.Command;
import org.apache.dolphinscheduler.dao.mapper.CommandMapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Date;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ThreadPoolExecutor;
/**
* DataSourceTest
*
*/
public class DataSourceTest {
private static final Logger logger = LoggerFactory.getLogger(DataSourceTest.class);
/**
* datasource test
* @param args args
*/
public static void main(String[] args) throws InterruptedException {
int threadNumber = 8;
int totalTests = 10;
ThreadPoolExecutor insertThreadPool = (ThreadPoolExecutor) ThreadUtils.newDaemonFixedThreadExecutor("insert-test-thread", threadNumber);
Date s = new Date();
for(int j = 0; j < totalTests; j++) {
Date start = new Date();
int commandCount = 1000;
CountDownLatch latch = new CountDownLatch(commandCount);
CommandMapper commandMapper = ConnectionFactory.getInstance().getMapper(CommandMapper.class);
for (int i = 0; i < commandCount; i++) {
Command command = new Command();
command.setId(i*j);
command.setProcessDefinitionCode(i*j);
command.setCommandType(CommandType.START_PROCESS);
command.setCommandParam("{\\\"ProcessInstanceId\\\":2}");
InsertThread insertThread = new InsertThread(commandMapper, command, latch);
insertThreadPool.execute(insertThread);
}
latch.await();
Date end = new Date();
logger.info(String.format("insert cost %d ms: thread : %d ,count: %d ", end.getTime() - start.getTime(), threadNumber, commandCount));
}
Date e = new Date();
logger.info(String.format("total cost %d ms, avg cost %s ms, count: %d ", e.getTime() - s.getTime(),String.valueOf((e.getTime() - s.getTime())/(totalTests * 1.0)), totalTests));
}
private static class InsertThread implements Runnable {
private CommandMapper commandMapper;
private Command command;
private CountDownLatch latch;
public InsertThread(CommandMapper commandMapper, Command command,CountDownLatch latch){
this.commandMapper = commandMapper;
this.command = command;
this.latch = latch;
}
@Override
public void run() {
Date start = new Date();
this.commandMapper.insert(command);
latch.countDown();
}
}
}
```
I test many times, the results like this:
```
Druid datasource
[INFO] 2021-10-05 17:48:31.782 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 3424 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 17:48:32.070 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 288 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 17:48:32.299 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 229 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 17:48:32.499 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 200 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 17:48:32.707 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 208 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 17:48:32.959 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 251 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 17:48:33.160 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 200 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 17:48:33.377 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 217 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 17:48:33.631 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 254 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 17:48:33.883 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 252 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 17:48:33.883 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[74] - total cost 5526 ms, avg cost 552.6 ms, count: 10
Hikari DataSource
[INFO] 2021-10-05 19:25:22.536 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 2906 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 19:25:22.720 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 184 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 19:25:22.944 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 224 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 19:25:23.085 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 141 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 19:25:23.237 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 152 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 19:25:23.372 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 135 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 19:25:23.479 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 107 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 19:25:23.608 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 128 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 19:25:23.709 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 101 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 19:25:23.815 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[70] - insert cost 105 ms: thread : 8 ,count: 1000
[INFO] 2021-10-05 19:25:23.815 org.apache.dolphinscheduler.dao.upgrade.DataSourceTest:[74] - total cost 4186 ms, avg cost 418.6 ms, count: 10
```
In the process of inserting data to the command table, we can see that HiKariCP DataSource is indeed better than Druid.
besides, HiKariCP is officially supported by Spring Boot 2+ and has better compatibility with Spring Boot
so I suggest we use HiKariCP instead of Druid, Do you have any suggestions? </div>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] dailidong commented on issue #6881: Change DataSource Connection Pool from Druid to HiKariCP
Posted by GitBox <gi...@apache.org>.
dailidong commented on issue #6881:
URL: https://github.com/apache/dolphinscheduler/issues/6881#issuecomment-971264013
#6828 has done this feature, I will close this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] dailidong closed issue #6881: Change DataSource Connection Pool from Druid to HiKariCP
Posted by GitBox <gi...@apache.org>.
dailidong closed issue #6881:
URL: https://github.com/apache/dolphinscheduler/issues/6881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] removed a comment on issue #6881: Change DataSource Connection Pool from Druid to HiKariCP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] removed a comment on issue #6881:
URL: https://github.com/apache/dolphinscheduler/issues/6881#issuecomment-971253693
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #6881: Change DataSource Connection Pool from Druid to HiKariCP
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #6881:
URL: https://github.com/apache/dolphinscheduler/issues/6881#issuecomment-971253693
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org