You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Jibing-Li (via GitHub)" <gi...@apache.org> on 2023/11/27 07:54:37 UTC
[PR] [Fix](statistics)Fix bug and improve auto analyze. [doris]
Jibing-Li opened a new pull request, #27626:
URL: https://github.com/apache/doris/pull/27626
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](statistics)Fix bug and improve auto analyze. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27626:
URL: https://github.com/apache/doris/pull/27626#issuecomment-1827740337
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.58 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 27 seconds loaded 2358488459 Bytes, about 83 MB/s
stream load orc: 70 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17099259840 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](statistics)Fix bug and improve auto analyze. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on code in PR #27626:
URL: https://github.com/apache/doris/pull/27626#discussion_r1405902956
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java:
##########
@@ -906,6 +906,16 @@ public static long getHugeTableAutoAnalyzeIntervalInMillis() {
return StatisticConstants.HUGE_TABLE_AUTO_ANALYZE_INTERVAL_IN_MILLIS;
}
+ public static long getExternalTableAutoAnalyzeIntervalInMillis() {
+ try {
+ return findConfigFromGlobalSessionVar(SessionVariable.EXTERNAL_TABLE_AUTO_ANALYZE_INTERVAL_IN_MILLIS)
+ .externalTableAutoAnalyzeIntervalInMillis;
+ } catch (Exception e) {
+ LOG.warn("Failed to get value of externalTableAutoAnalyzeIntervalInMillis, return default", e);
Review Comment:
I didn't dive deep in this, I simply followed all the other variables' format. Seems like VariableMgr.getValue may throw exception when the session name not exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](statistics)Fix bug and improve auto analyze. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27626:
URL: https://github.com/apache/doris/pull/27626#issuecomment-1827741969
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit dd9b6b3663e647c26516b52d4992a0ef17cd72d6, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4916 4630 4653 4630
q2 363 154 157 154
q3 2067 1978 1882 1882
q4 1392 1287 1261 1261
q5 3955 3953 4009 3953
q6 255 136 129 129
q7 1414 877 879 877
q8 2782 2802 2762 2762
q9 9880 9663 10827 9663
q10 3449 3546 3514 3514
q11 372 240 248 240
q12 437 288 302 288
q13 4532 3852 3791 3791
q14 320 282 290 282
q15 591 534 518 518
q16 660 584 583 583
q17 1134 964 920 920
q18 7885 7431 7631 7431
q19 1679 1670 1660 1660
q20 586 289 312 289
q21 4477 3999 4022 3999
q22 476 387 403 387
Total cold run time: 53622 ms
Total hot run time: 49213 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4594 4593 4576 4576
q2 340 235 246 235
q3 4061 3984 4027 3984
q4 2721 2697 2710 2697
q5 9658 9706 9630 9630
q6 245 125 126 125
q7 3071 2488 2507 2488
q8 4456 4474 4476 4474
q9 13299 13146 13211 13146
q10 4061 4136 4185 4136
q11 787 659 735 659
q12 965 797 812 797
q13 4311 3592 3571 3571
q14 396 350 346 346
q15 581 519 524 519
q16 732 692 702 692
q17 3851 3906 3882 3882
q18 9653 9239 9223 9223
q19 1817 1791 1806 1791
q20 2417 2061 2024 2024
q21 8859 8578 8839 8578
q22 897 779 843 779
Total cold run time: 81772 ms
Total hot run time: 78352 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](statistics)Fix bug and improve auto analyze. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on code in PR #27626:
URL: https://github.com/apache/doris/pull/27626#discussion_r1405934855
##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -1366,6 +1369,12 @@ public void setEnableLeftZigZag(boolean enableLeftZigZag) {
+ "tables larger than huge_table_lower_bound_size_in_bytes are analyzed only once."})
public long hugeTableAutoAnalyzeIntervalInMillis = TimeUnit.HOURS.toMillis(12);
+ @VariableMgr.VarAttr(name = EXTERNAL_TABLE_AUTO_ANALYZE_INTERVAL_IN_MILLIS, flag = VariableMgr.GLOBAL,
+ description = {"控制对外表的自动ANALYZE的最小时间间隔，在该时间间隔内的外表仅ANALYZE一次",
+ "This controls the minimum time interval for automatic ANALYZE on external tables."
+ + "Within this interval, external tables are analyzed only once."})
+ public long externalTableAutoAnalyzeIntervalInMillis = TimeUnit.HOURS.toMillis(240);
Review Comment:
changed to 1 day
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](statistics)Fix bug and improve auto analyze. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27626:
URL: https://github.com/apache/doris/pull/27626#issuecomment-1827914136
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](statistics)Fix bug and improve auto analyze. [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #27626:
URL: https://github.com/apache/doris/pull/27626
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](statistics)Fix bug and improve auto analyze. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27626:
URL: https://github.com/apache/doris/pull/27626#issuecomment-1827825758
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](statistics)Fix bug and improve auto analyze. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #27626:
URL: https://github.com/apache/doris/pull/27626#issuecomment-1827430903
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](statistics)Fix bug and improve auto analyze. [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #27626:
URL: https://github.com/apache/doris/pull/27626#discussion_r1405887622
##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -1366,6 +1369,12 @@ public void setEnableLeftZigZag(boolean enableLeftZigZag) {
+ "tables larger than huge_table_lower_bound_size_in_bytes are analyzed only once."})
public long hugeTableAutoAnalyzeIntervalInMillis = TimeUnit.HOURS.toMillis(12);
+ @VariableMgr.VarAttr(name = EXTERNAL_TABLE_AUTO_ANALYZE_INTERVAL_IN_MILLIS, flag = VariableMgr.GLOBAL,
+ description = {"控制对外表的自动ANALYZE的最小时间间隔，在该时间间隔内的外表仅ANALYZE一次",
+ "This controls the minimum time interval for automatic ANALYZE on external tables."
+ + "Within this interval, external tables are analyzed only once."})
+ public long externalTableAutoAnalyzeIntervalInMillis = TimeUnit.HOURS.toMillis(240);
Review Comment:
10 days is too long
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java:
##########
@@ -906,6 +906,16 @@ public static long getHugeTableAutoAnalyzeIntervalInMillis() {
return StatisticConstants.HUGE_TABLE_AUTO_ANALYZE_INTERVAL_IN_MILLIS;
}
+ public static long getExternalTableAutoAnalyzeIntervalInMillis() {
+ try {
+ return findConfigFromGlobalSessionVar(SessionVariable.EXTERNAL_TABLE_AUTO_ANALYZE_INTERVAL_IN_MILLIS)
+ .externalTableAutoAnalyzeIntervalInMillis;
+ } catch (Exception e) {
+ LOG.warn("Failed to get value of externalTableAutoAnalyzeIntervalInMillis, return default", e);
Review Comment:
Why is an exception thrown here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Fix](statistics)Fix bug and improve auto analyze. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #27626:
URL: https://github.com/apache/doris/pull/27626#issuecomment-1827545092
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org