You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Kikyou1997 (via GitHub)" <gi...@apache.org> on 2023/11/27 09:16:18 UTC
[PR] [enhancement](stats) Stats auto collector execute in serial [doris]
Kikyou1997 opened a new pull request, #27634:
URL: https://github.com/apache/doris/pull/27634
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](stats) Stats auto collector execute in serial [doris]
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on code in PR #27634:
URL: https://github.com/apache/doris/pull/27634#discussion_r1405885284
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java:
##########
@@ -83,7 +74,14 @@ protected void createSystemAnalysisJob(AnalysisInfo jobInfo)
analysisManager.createTableLevelTaskForExternalTable(jobInfo, analysisTasks, false);
}
Env.getCurrentEnv().getAnalysisManager().registerSysJob(jobInfo, analysisTasks);
- analysisTasks.values().forEach(analysisTaskExecutor::submitTask);
+ for (BaseAnalysisTask task : analysisTasks.values()) {
+ try {
+ task.execute();
Review Comment:
Yes, I agree with you and the logic you mentioned has been implemented in `AnalysisJob`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](stats) Stats auto collector execute in serial [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27634:
URL: https://github.com/apache/doris/pull/27634#issuecomment-1827457672
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](stats) Stats auto collector execute in serial [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow closed pull request #27634: [enhancement](stats) Stats auto collector execute in serial
URL: https://github.com/apache/doris/pull/27634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](stats) Stats auto collector execute in serial [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on code in PR #27634:
URL: https://github.com/apache/doris/pull/27634#discussion_r1405874445
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java:
##########
@@ -83,7 +74,14 @@ protected void createSystemAnalysisJob(AnalysisInfo jobInfo)
analysisManager.createTableLevelTaskForExternalTable(jobInfo, analysisTasks, false);
}
Env.getCurrentEnv().getAnalysisManager().registerSysJob(jobInfo, analysisTasks);
- analysisTasks.values().forEach(analysisTaskExecutor::submitTask);
+ for (BaseAnalysisTask task : analysisTasks.values()) {
+ try {
+ task.execute();
Review Comment:
Not a block. But it's better to write the successfully executed tasks' result to column statistics table, instead of fail the whole job when one task failed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](stats) Stats auto collector execute in serial [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27634:
URL: https://github.com/apache/doris/pull/27634#issuecomment-1827648590
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit b29546b5a34f1a30c18e0f50f2ee5763e6a0489b, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4899 4703 4665 4665
q2 366 162 144 144
q3 2044 1944 1957 1944
q4 1413 1291 1276 1276
q5 4004 3989 4015 3989
q6 258 138 137 137
q7 1425 876 886 876
q8 2803 2816 2783 2783
q9 9809 9643 9603 9603
q10 3455 3643 3519 3519
q11 386 256 261 256
q12 440 290 290 290
q13 4612 3837 3846 3837
q14 325 300 301 300
q15 588 532 523 523
q16 664 588 592 588
q17 1146 979 940 940
q18 7901 7393 7595 7393
q19 1680 1667 1691 1667
q20 577 319 323 319
q21 4444 4012 4032 4012
q22 473 378 382 378
Total cold run time: 53712 ms
Total hot run time: 49439 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4582 4585 4574 4574
q2 339 239 260 239
q3 4036 4011 4018 4011
q4 2726 2730 2716 2716
q5 9609 9555 9604 9555
q6 249 126 126 126
q7 3063 2510 2486 2486
q8 4484 4442 4429 4429
q9 13200 13119 13166 13119
q10 4070 4159 4177 4159
q11 766 675 638 638
q12 988 806 819 806
q13 4271 3618 3584 3584
q14 375 357 364 357
q15 571 526 524 524
q16 746 686 687 686
q17 3911 3843 3865 3843
q18 9751 9146 9042 9042
q19 1832 1840 1775 1775
q20 2402 2066 2048 2048
q21 8977 8515 8819 8515
q22 902 842 790 790
Total cold run time: 81850 ms
Total hot run time: 78022 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](stats) Stats auto collector execute in serial [doris]
Posted by "Kikyou1997 (via GitHub)" <gi...@apache.org>.
Kikyou1997 commented on PR #27634:
URL: https://github.com/apache/doris/pull/27634#issuecomment-1827437873
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](stats) Stats auto collector execute in serial [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27634:
URL: https://github.com/apache/doris/pull/27634#issuecomment-1827638899
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.09 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 26 seconds loaded 2358488459 Bytes, about 86 MB/s
stream load orc: 71 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17100488403 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org