You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Jibing-Li (via GitHub)" <gi...@apache.org> on 2024/03/10 06:09:26 UTC
[PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Jibing-Li opened a new pull request, #32030:
URL: https://github.com/apache/doris/pull/32030
If getRowCount returns 0, and table visible version is 2, the table is probably not empty, so we wait a moment for the row count to be reported.
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32030:
URL: https://github.com/apache/doris/pull/32030#issuecomment-1987109838
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 72b81cbaf19b39bed4197be57f46903f3f3625c5 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 17.4 seconds inserted 10000000 Rows, about 574K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #32030:
URL: https://github.com/apache/doris/pull/32030#discussion_r1520809331
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java:
##########
@@ -974,4 +976,28 @@ public static boolean isMvColumn(TableIf table, String columnName) {
|| columnName.startsWith(CreateMaterializedViewStmt.MATERIALIZED_VIEW_AGGREGATE_NAME_PREFIX);
}
+ public static boolean isEmptyTable(TableIf table, AnalysisInfo.AnalysisMethod method) {
+ int waitRowCountReportedTime = 90;
+ if (!(table instanceof OlapTable) || method.equals(AnalysisInfo.AnalysisMethod.FULL)) {
+ return false;
+ }
+ OlapTable olapTable = (OlapTable) table;
+ for (int i = 0; i < waitRowCountReportedTime; i++) {
+ if (olapTable.getRowCount() > 0) {
+ return false;
+ }
+ // If visible version is 2, table is probably not empty. So we wait row count to be reported.
+ // If visible version is not 2 and getRowCount return 0, we assume it is an empty table.
+ if (olapTable.getVisibleVersion() != TableAttributes.TABLE_INIT_VERSION + 1) {
+ return true;
+ }
+ try {
+ Thread.sleep(1000);
Review Comment:
A delay for auto analysis is acceptable, but what if it is a manual one with sync? Queuing in the task queue will make users wait longer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32030:
URL: https://github.com/apache/doris/pull/32030#issuecomment-1990069741
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #32030:
URL: https://github.com/apache/doris/pull/32030#issuecomment-1987103531
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #32030:
URL: https://github.com/apache/doris/pull/32030#discussion_r1522417524
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java:
##########
@@ -974,4 +976,28 @@ public static boolean isMvColumn(TableIf table, String columnName) {
|| columnName.startsWith(CreateMaterializedViewStmt.MATERIALIZED_VIEW_AGGREGATE_NAME_PREFIX);
}
+ public static boolean isEmptyTable(TableIf table, AnalysisInfo.AnalysisMethod method) {
+ int waitRowCountReportedTime = 90;
+ if (!(table instanceof OlapTable) || method.equals(AnalysisInfo.AnalysisMethod.FULL)) {
+ return false;
+ }
+ OlapTable olapTable = (OlapTable) table;
+ for (int i = 0; i < waitRowCountReportedTime; i++) {
+ if (olapTable.getRowCount() > 0) {
+ return false;
+ }
+ // If visible version is 2, table is probably not empty. So we wait row count to be reported.
+ // If visible version is not 2 and getRowCount return 0, we assume it is an empty table.
+ if (olapTable.getVisibleVersion() != TableAttributes.TABLE_INIT_VERSION + 1) {
+ return true;
+ }
+ try {
+ Thread.sleep(1000);
+ } catch (InterruptedException e) {
+ LOG.info("Sleep interrupted.", e);
Review Comment:
why print this log?
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java:
##########
@@ -974,4 +976,28 @@ public static boolean isMvColumn(TableIf table, String columnName) {
|| columnName.startsWith(CreateMaterializedViewStmt.MATERIALIZED_VIEW_AGGREGATE_NAME_PREFIX);
}
+ public static boolean isEmptyTable(TableIf table, AnalysisInfo.AnalysisMethod method) {
+ int waitRowCountReportedTime = 90;
Review Comment:
nit: should be static final variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #32030:
URL: https://github.com/apache/doris/pull/32030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on code in PR #32030:
URL: https://github.com/apache/doris/pull/32030#discussion_r1520810453
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java:
##########
@@ -974,4 +976,28 @@ public static boolean isMvColumn(TableIf table, String columnName) {
|| columnName.startsWith(CreateMaterializedViewStmt.MATERIALIZED_VIEW_AGGREGATE_NAME_PREFIX);
}
+ public static boolean isEmptyTable(TableIf table, AnalysisInfo.AnalysisMethod method) {
+ int waitRowCountReportedTime = 90;
+ if (!(table instanceof OlapTable) || method.equals(AnalysisInfo.AnalysisMethod.FULL)) {
+ return false;
+ }
+ OlapTable olapTable = (OlapTable) table;
+ for (int i = 0; i < waitRowCountReportedTime; i++) {
+ if (olapTable.getRowCount() > 0) {
+ return false;
+ }
+ // If visible version is 2, table is probably not empty. So we wait row count to be reported.
+ // If visible version is not 2 and getRowCount return 0, we assume it is an empty table.
+ if (olapTable.getVisibleVersion() != TableAttributes.TABLE_INIT_VERSION + 1) {
+ return true;
+ }
+ try {
+ Thread.sleep(1000);
Review Comment:
Each manual analyze with sync operation will create its own thread pool, so it will not block other session.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #32030:
URL: https://github.com/apache/doris/pull/32030#discussion_r1519646289
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java:
##########
@@ -974,4 +976,28 @@ public static boolean isMvColumn(TableIf table, String columnName) {
|| columnName.startsWith(CreateMaterializedViewStmt.MATERIALIZED_VIEW_AGGREGATE_NAME_PREFIX);
}
+ public static boolean isEmptyTable(TableIf table, AnalysisInfo.AnalysisMethod method) {
+ int waitRowCountReportedTime = 90;
+ if (!(table instanceof OlapTable) || method.equals(AnalysisInfo.AnalysisMethod.FULL)) {
+ return false;
+ }
+ OlapTable olapTable = (OlapTable) table;
+ for (int i = 0; i < waitRowCountReportedTime; i++) {
+ if (olapTable.getRowCount() > 0) {
+ return false;
+ }
+ // If visible version is 2, table is probably not empty. So we wait row count to be reported.
+ // If visible version is not 2 and getRowCount return 0, we assume it is an empty table.
+ if (olapTable.getVisibleVersion() != TableAttributes.TABLE_INIT_VERSION + 1) {
+ return true;
+ }
+ try {
+ Thread.sleep(1000);
Review Comment:
Will this waiting hold up the Worker thread? AFAIK, number of threads is limited in the analysisTaskExecutor.
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java:
##########
@@ -974,4 +976,28 @@ public static boolean isMvColumn(TableIf table, String columnName) {
|| columnName.startsWith(CreateMaterializedViewStmt.MATERIALIZED_VIEW_AGGREGATE_NAME_PREFIX);
}
+ public static boolean isEmptyTable(TableIf table, AnalysisInfo.AnalysisMethod method) {
+ int waitRowCountReportedTime = 90;
+ if (!(table instanceof OlapTable) || method.equals(AnalysisInfo.AnalysisMethod.FULL)) {
+ return false;
+ }
+ OlapTable olapTable = (OlapTable) table;
+ for (int i = 0; i < waitRowCountReportedTime; i++) {
+ if (olapTable.getRowCount() > 0) {
+ return false;
+ }
+ // If visible version is 2, table is probably not empty. So we wait row count to be reported.
Review Comment:
Where does this assumption come from and why it is helpful to wait?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32030:
URL: https://github.com/apache/doris/pull/32030#issuecomment-1987106497
<details>
<summary>TPC-H: <b>Total hot run time: 38185 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72b81cbaf19b39bed4197be57f46903f3f3625c5, data reload: false
------ Round 1 ----------------------------------
q1 18800 4204 4118 4118
q2 2461 159 146 146
q3 11193 966 968 966
q4 4817 963 965 963
q5 8031 2929 3125 2929
q6 183 126 128 126
q7 1335 864 845 845
q8 9896 2131 2094 2094
q9 7287 6476 6429 6429
q10 8212 2572 2556 2556
q11 421 227 217 217
q12 736 317 309 309
q13 17954 2957 2937 2937
q14 290 243 258 243
q15 485 443 447 443
q16 479 394 406 394
q17 949 895 850 850
q18 6759 5881 5900 5881
q19 1589 1527 1522 1522
q20 554 280 274 274
q21 7566 3651 3710 3651
q22 769 292 327 292
Total cold run time: 110766 ms
Total hot run time: 38185 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4016 3978 4003 3978
q2 319 221 235 221
q3 2957 2917 2921 2917
q4 1847 1798 1776 1776
q5 5215 5242 5200 5200
q6 204 117 122 117
q7 2252 1859 1815 1815
q8 3228 3268 3326 3268
q9 8515 8536 8510 8510
q10 6172 3731 3703 3703
q11 526 463 456 456
q12 689 540 548 540
q13 5908 2825 2809 2809
q14 273 260 260 260
q15 471 441 443 441
q16 479 412 416 412
q17 1697 1626 1642 1626
q18 8023 7403 7185 7185
q19 1636 1643 1622 1622
q20 1953 1727 1746 1727
q21 4799 4840 4822 4822
q22 524 460 462 460
Total cold run time: 61703 ms
Total hot run time: 53865 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32030:
URL: https://github.com/apache/doris/pull/32030#issuecomment-1987103439
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32030:
URL: https://github.com/apache/doris/pull/32030#issuecomment-1993230818
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32030:
URL: https://github.com/apache/doris/pull/32030#issuecomment-1987109286
<details>
<summary>ClickBench: <b>Total hot run time: 30.57 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 72b81cbaf19b39bed4197be57f46903f3f3625c5, data reload: false
query1 0.05 0.03 0.04
query2 0.07 0.02 0.02
query3 0.23 0.07 0.06
query4 1.64 0.10 0.09
query5 0.48 0.48 0.49
query6 1.27 0.67 0.66
query7 0.02 0.01 0.02
query8 0.05 0.03 0.03
query9 0.57 0.51 0.49
query10 0.56 0.56 0.56
query11 0.13 0.10 0.10
query12 0.12 0.10 0.11
query13 0.57 0.57 0.56
query14 0.74 0.78 0.75
query15 0.81 0.80 0.80
query16 0.38 0.37 0.37
query17 1.01 1.00 0.96
query18 0.26 0.26 0.26
query19 1.79 1.73 1.72
query20 0.02 0.01 0.01
query21 15.44 0.67 0.62
query22 2.86 4.94 2.23
query23 17.43 1.03 0.92
query24 2.28 0.42 0.39
query25 0.25 0.05 0.06
query26 0.17 0.14 0.14
query27 0.03 0.02 0.02
query28 12.10 0.85 0.82
query29 12.52 3.52 3.27
query30 0.62 0.60 0.58
query31 2.80 0.34 0.35
query32 3.34 0.43 0.43
query33 2.94 2.94 2.84
query34 15.49 4.27 4.31
query35 4.29 4.28 4.30
query36 1.11 1.02 1.02
query37 0.08 0.05 0.05
query38 0.04 0.03 0.03
query39 0.03 0.02 0.02
query40 0.17 0.14 0.13
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.03 0.02 0.03
Total cold run time: 104.9 s
Total hot run time: 30.57 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32030:
URL: https://github.com/apache/doris/pull/32030#issuecomment-1993230788
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on code in PR #32030:
URL: https://github.com/apache/doris/pull/32030#discussion_r1519805761
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java:
##########
@@ -974,4 +976,28 @@ public static boolean isMvColumn(TableIf table, String columnName) {
|| columnName.startsWith(CreateMaterializedViewStmt.MATERIALIZED_VIEW_AGGREGATE_NAME_PREFIX);
}
+ public static boolean isEmptyTable(TableIf table, AnalysisInfo.AnalysisMethod method) {
+ int waitRowCountReportedTime = 90;
+ if (!(table instanceof OlapTable) || method.equals(AnalysisInfo.AnalysisMethod.FULL)) {
+ return false;
+ }
+ OlapTable olapTable = (OlapTable) table;
+ for (int i = 0; i < waitRowCountReportedTime; i++) {
+ if (olapTable.getRowCount() > 0) {
+ return false;
+ }
+ // If visible version is 2, table is probably not empty. So we wait row count to be reported.
+ // If visible version is not 2 and getRowCount return 0, we assume it is an empty table.
+ if (olapTable.getVisibleVersion() != TableAttributes.TABLE_INIT_VERSION + 1) {
+ return true;
+ }
+ try {
+ Thread.sleep(1000);
Review Comment:
Yes, it will hold up the worker thread. But I don't think it's a problem. Because in most case, we don't need to wait, only when the visible version is 2 and get row count == 0, and the waiting time is usually less than 1 minute. Also, couple of minutes delay for auto analyze is acceptable.
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java:
##########
@@ -974,4 +976,28 @@ public static boolean isMvColumn(TableIf table, String columnName) {
|| columnName.startsWith(CreateMaterializedViewStmt.MATERIALIZED_VIEW_AGGREGATE_NAME_PREFIX);
}
+ public static boolean isEmptyTable(TableIf table, AnalysisInfo.AnalysisMethod method) {
+ int waitRowCountReportedTime = 90;
+ if (!(table instanceof OlapTable) || method.equals(AnalysisInfo.AnalysisMethod.FULL)) {
+ return false;
+ }
+ OlapTable olapTable = (OlapTable) table;
+ for (int i = 0; i < waitRowCountReportedTime; i++) {
+ if (olapTable.getRowCount() > 0) {
+ return false;
+ }
+ // If visible version is 2, table is probably not empty. So we wait row count to be reported.
Review Comment:
Table visible version == 2 usually means use loaded the first batch of data to the table, in this case, the table is probably not empty. In this case, we wait the row count to be reported so that we can avoid get all 0 statistics. If visible version is 1 (init), the table is empty for sure. If it is > 2, we don't know it's empty or not, because of delete or truncate and so on. In this case we don't wait, simply use the row count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32030:
URL: https://github.com/apache/doris/pull/32030#issuecomment-1987108422
<details>
<summary>TPC-DS: <b>Total hot run time: 186233 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 72b81cbaf19b39bed4197be57f46903f3f3625c5, data reload: false
query1 942 368 339 339
query2 7358 2008 2098 2008
query3 6706 224 211 211
query4 27327 21321 21069 21069
query5 4363 472 506 472
query6 285 184 177 177
query7 4640 308 310 308
query8 240 175 171 171
query9 8426 2316 2302 2302
query10 466 244 264 244
query11 15124 14417 14387 14387
query12 142 92 87 87
query13 1634 428 446 428
query14 10946 9760 9621 9621
query15 240 193 201 193
query16 8031 274 286 274
query17 1719 586 567 567
query18 2065 284 282 282
query19 210 166 163 163
query20 97 90 87 87
query21 208 132 129 129
query22 4627 4453 4524 4453
query23 31407 30635 30421 30421
query24 11606 3069 3042 3042
query25 685 397 387 387
query26 1929 176 179 176
query27 2978 379 370 370
query28 6544 1882 1898 1882
query29 1109 602 617 602
query30 314 157 147 147
query31 941 772 756 756
query32 100 68 61 61
query33 757 268 269 268
query34 1007 465 485 465
query35 872 651 621 621
query36 930 780 821 780
query37 150 75 76 75
query38 3207 3094 3148 3094
query39 1412 1377 1378 1377
query40 295 125 123 123
query41 61 57 56 56
query42 111 115 102 102
query43 446 425 402 402
query44 1092 714 700 700
query45 279 228 232 228
query46 1056 806 800 800
query47 1619 1551 1543 1543
query48 423 354 362 354
query49 1226 351 336 336
query50 785 379 381 379
query51 6732 6752 6737 6737
query52 117 96 93 93
query53 360 293 293 293
query54 325 255 244 244
query55 87 92 87 87
query56 246 256 228 228
query57 1081 1003 989 989
query58 251 216 227 216
query59 2498 2346 2406 2346
query60 269 266 251 251
query61 96 92 109 92
query62 649 447 420 420
query63 303 278 283 278
query64 6102 3276 3094 3094
query65 3045 3015 3025 3015
query66 1443 350 324 324
query67 14985 14408 14423 14408
query68 12268 571 578 571
query69 737 384 379 379
query70 1362 1124 1087 1087
query71 630 284 267 267
query72 10069 2591 2462 2462
query73 1911 338 336 336
query74 7257 6764 6703 6703
query75 11744 7959 7971 7959
query76 7920 1070 1188 1070
query77 941 252 263 252
query78 10153 9528 9499 9499
query79 10715 532 524 524
query80 766 466 403 403
query81 481 215 205 205
query82 340 202 194 194
query83 239 144 145 144
query84 287 81 80 80
query85 1115 331 375 331
query86 380 288 289 288
query87 3394 3148 3166 3148
query88 3138 2378 2367 2367
query89 540 367 368 367
query90 2144 183 189 183
query91 172 155 143 143
query92 61 49 49 49
query93 3624 551 527 527
query94 1358 187 189 187
query95 456 354 357 354
query96 597 264 265 264
query97 4433 4297 4312 4297
query98 235 208 213 208
query99 1075 757 751 751
Total cold run time: 318268 ms
Total hot run time: 186233 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Add wait row count reported logic for sample analyze. [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #32030:
URL: https://github.com/apache/doris/pull/32030#discussion_r1519646289
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java:
##########
@@ -974,4 +976,28 @@ public static boolean isMvColumn(TableIf table, String columnName) {
|| columnName.startsWith(CreateMaterializedViewStmt.MATERIALIZED_VIEW_AGGREGATE_NAME_PREFIX);
}
+ public static boolean isEmptyTable(TableIf table, AnalysisInfo.AnalysisMethod method) {
+ int waitRowCountReportedTime = 90;
+ if (!(table instanceof OlapTable) || method.equals(AnalysisInfo.AnalysisMethod.FULL)) {
+ return false;
+ }
+ OlapTable olapTable = (OlapTable) table;
+ for (int i = 0; i < waitRowCountReportedTime; i++) {
+ if (olapTable.getRowCount() > 0) {
+ return false;
+ }
+ // If visible version is 2, table is probably not empty. So we wait row count to be reported.
+ // If visible version is not 2 and getRowCount return 0, we assume it is an empty table.
+ if (olapTable.getVisibleVersion() != TableAttributes.TABLE_INIT_VERSION + 1) {
+ return true;
+ }
+ try {
+ Thread.sleep(1000);
Review Comment:
Will this waiting hold up the Worker thread? AFAIK, the number of threads is quite limited in the analysisTaskExecutor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org