You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Jibing-Li (via GitHub)" <gi...@apache.org> on 2024/02/21 07:59:08 UTC
[PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Jibing-Li opened a new pull request, #31212:
URL: https://github.com/apache/doris/pull/31212
Release useless StmtExecutor reference in time to reduce memory useage.
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #31212:
URL: https://github.com/apache/doris/pull/31212#discussion_r1497068087
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java:
##########
@@ -183,14 +184,22 @@ protected ResultRow collectBasicStat(AutoCloseConnectContext context) {
tbl.getName(), col.getName());
return null;
}
+ long startTime = System.currentTimeMillis();
Map<String, String> params = new HashMap<>();
params.put("dbName", db.getFullName());
params.put("colName", StatisticsUtil.escapeColumnName(info.colName));
params.put("tblName", tbl.getName());
params.put("index", getIndex());
StringSubstitutor stringSubstitutor = new StringSubstitutor(params);
- stmtExecutor = new StmtExecutor(context.connectContext, stringSubstitutor.replace(BASIC_STATS_TEMPLATE));
- return stmtExecutor.executeInternalQuery().get(0);
+ String sql = stringSubstitutor.replace(BASIC_STATS_TEMPLATE);
+ stmtExecutor = new StmtExecutor(context.connectContext, sql);
+ ResultRow resultRow = stmtExecutor.executeInternalQuery().get(0);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Cost time in millisec: " + (System.currentTimeMillis() - startTime)
+ + " Min max SQL: " + sql + " QueryId: " + DebugUtil.printId(stmtExecutor.getContext().queryId()));
+ }
+ stmtExecutor = null;
Review Comment:
add some comment to explain why need set to null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31212:
URL: https://github.com/apache/doris/pull/31212#issuecomment-1956080672
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31212:
URL: https://github.com/apache/doris/pull/31212#issuecomment-1956375427
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on code in PR #31212:
URL: https://github.com/apache/doris/pull/31212#discussion_r1497091875
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java:
##########
@@ -183,14 +184,22 @@ protected ResultRow collectBasicStat(AutoCloseConnectContext context) {
tbl.getName(), col.getName());
return null;
}
+ long startTime = System.currentTimeMillis();
Map<String, String> params = new HashMap<>();
params.put("dbName", db.getFullName());
params.put("colName", StatisticsUtil.escapeColumnName(info.colName));
params.put("tblName", tbl.getName());
params.put("index", getIndex());
StringSubstitutor stringSubstitutor = new StringSubstitutor(params);
- stmtExecutor = new StmtExecutor(context.connectContext, stringSubstitutor.replace(BASIC_STATS_TEMPLATE));
- return stmtExecutor.executeInternalQuery().get(0);
+ String sql = stringSubstitutor.replace(BASIC_STATS_TEMPLATE);
+ stmtExecutor = new StmtExecutor(context.connectContext, sql);
+ ResultRow resultRow = stmtExecutor.executeInternalQuery().get(0);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Cost time in millisec: " + (System.currentTimeMillis() - startTime)
+ + " Min max SQL: " + sql + " QueryId: " + DebugUtil.printId(stmtExecutor.getContext().queryId()));
+ }
+ stmtExecutor = null;
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31212:
URL: https://github.com/apache/doris/pull/31212#issuecomment-1956192673
<details>
<summary>ClickBench: <b>Total hot run time: 30.01 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 468d24f4abf4a5fb9425a81b20e0c5f5bdd33d8c, data reload: false
query1 0.03 0.02 0.02
query2 0.07 0.03 0.02
query3 0.22 0.07 0.08
query4 1.64 0.09 0.08
query5 0.49 0.48 0.48
query6 1.36 0.62 0.64
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.52 0.44 0.45
query10 0.51 0.47 0.47
query11 0.13 0.11 0.09
query12 0.12 0.10 0.10
query13 0.60 0.59 0.58
query14 0.76 0.79 0.77
query15 0.82 0.80 0.79
query16 0.32 0.33 0.34
query17 0.92 0.89 0.87
query18 0.18 0.19 0.16
query19 1.74 1.70 1.67
query20 0.01 0.02 0.01
query21 15.40 0.64 0.61
query22 3.47 3.88 1.64
query23 17.20 1.04 1.00
query24 2.02 0.94 0.12
query25 0.50 0.07 0.06
query26 0.16 0.14 0.14
query27 0.05 0.05 0.06
query28 11.96 0.84 0.82
query29 12.64 3.28 3.38
query30 0.55 0.49 0.49
query31 2.80 0.36 0.36
query32 3.34 0.47 0.48
query33 3.17 3.16 3.10
query34 15.36 4.51 4.53
query35 4.55 4.54 4.53
query36 1.06 0.95 0.94
query37 0.07 0.05 0.06
query38 0.04 0.03 0.03
query39 0.02 0.02 0.02
query40 0.16 0.16 0.14
query41 0.07 0.02 0.01
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 105.13 s
Total hot run time: 30.01 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31212:
URL: https://github.com/apache/doris/pull/31212#issuecomment-1956164677
<details>
<summary>TPC-H: <b>Total hot run time: 41905 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 468d24f4abf4a5fb9425a81b20e0c5f5bdd33d8c, data reload: false
------ Round 1 ----------------------------------
q1 17631 5036 4923 4923
q2 2051 156 140 140
q3 10567 1033 1067 1033
q4 4655 998 997 997
q5 7719 3241 3296 3241
q6 193 132 132 132
q7 1263 815 779 779
q8 9340 2111 2107 2107
q9 7767 6771 6763 6763
q10 8379 2654 2648 2648
q11 423 238 212 212
q12 785 335 338 335
q13 18029 3686 3703 3686
q14 294 274 262 262
q15 621 508 500 500
q16 517 408 441 408
q17 953 875 856 856
q18 7616 6703 6685 6685
q19 1551 1508 1511 1508
q20 629 366 360 360
q21 6747 3985 4032 3985
q22 865 351 345 345
Total cold run time: 108595 ms
Total hot run time: 41905 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4866 4894 4896 4894
q2 292 184 192 184
q3 3635 3614 3604 3604
q4 2550 2558 2576 2558
q5 5823 5809 5827 5809
q6 210 128 129 128
q7 2272 1647 1679 1647
q8 3056 3137 3143 3137
q9 8866 8812 8767 8767
q10 6782 4256 4264 4256
q11 517 363 369 363
q12 781 570 557 557
q13 4421 3425 3412 3412
q14 268 246 222 222
q15 601 534 498 498
q16 470 443 424 424
q17 1735 1634 1641 1634
q18 8382 7681 7628 7628
q19 1644 1643 1652 1643
q20 2123 1847 1842 1842
q21 6670 6291 6243 6243
q22 586 511 510 510
Total cold run time: 66550 ms
Total hot run time: 59960 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31212:
URL: https://github.com/apache/doris/pull/31212#issuecomment-1956375500
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #31212:
URL: https://github.com/apache/doris/pull/31212#issuecomment-1956131680
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31212:
URL: https://github.com/apache/doris/pull/31212#issuecomment-1956198839
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 468d24f4abf4a5fb9425a81b20e0c5f5bdd33d8c with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.6 seconds inserted 10000000 Rows, about 735K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31212:
URL: https://github.com/apache/doris/pull/31212#issuecomment-1956183302
<details>
<summary>TPC-DS: <b>Total hot run time: 178564 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 468d24f4abf4a5fb9425a81b20e0c5f5bdd33d8c, data reload: false
query1 873 352 350 350
query2 5361 1855 1835 1835
query3 5480 206 203 203
query4 22841 21153 21160 21153
query5 3648 374 376 374
query6 246 175 168 168
query7 3600 292 294 292
query8 223 207 208 207
query9 6438 2888 2878 2878
query10 328 227 215 215
query11 15078 14569 14491 14491
query12 128 84 82 82
query13 1192 427 429 427
query14 8501 7779 7756 7756
query15 200 197 192 192
query16 5906 259 252 252
query17 784 558 550 550
query18 1221 270 265 265
query19 192 144 149 144
query20 88 81 88 81
query21 175 121 131 121
query22 5019 4998 4813 4813
query23 32501 31556 31598 31556
query24 10184 3385 3339 3339
query25 581 377 364 364
query26 983 159 157 157
query27 2140 319 309 309
query28 5445 1840 1837 1837
query29 1020 622 628 622
query30 284 139 149 139
query31 904 768 766 766
query32 67 62 57 57
query33 645 248 235 235
query34 846 502 500 500
query35 957 840 844 840
query36 972 918 862 862
query37 205 60 66 60
query38 3293 3194 3267 3194
query39 1371 1335 1311 1311
query40 170 109 104 104
query41 39 36 36 36
query42 115 100 102 100
query43 486 456 454 454
query44 1019 695 707 695
query45 199 175 175 175
query46 1055 775 750 750
query47 1617 1564 1504 1504
query48 423 343 347 343
query49 769 317 302 302
query50 790 368 374 368
query51 5228 5177 5184 5177
query52 107 94 92 92
query53 391 306 300 300
query54 333 228 229 228
query55 81 80 79 79
query56 221 206 203 203
query57 1054 956 997 956
query58 222 209 198 198
query59 2409 2293 2294 2293
query60 246 216 238 216
query61 91 88 89 88
query62 614 375 372 372
query63 330 287 293 287
query64 7103 3117 3143 3117
query65 3270 3260 3253 3253
query66 1571 329 318 318
query67 14858 14233 14140 14140
query68 3742 559 544 544
query69 494 362 362 362
query70 1233 1188 1248 1188
query71 312 254 258 254
query72 6453 2875 2671 2671
query73 697 318 316 316
query74 6881 6563 6545 6545
query75 3166 2600 2584 2584
query76 2684 1111 1216 1111
query77 351 237 238 237
query78 9414 8888 8865 8865
query79 963 500 515 500
query80 525 357 344 344
query81 403 205 215 205
query82 242 83 90 83
query83 144 123 120 120
query84 250 79 79 79
query85 782 363 362 362
query86 301 292 300 292
query87 3492 3418 3297 3297
query88 2698 2309 2301 2301
query89 437 349 341 341
query90 1809 164 160 160
query91 161 134 133 133
query92 52 49 51 49
query93 1000 514 505 505
query94 1065 177 184 177
query95 483 368 378 368
query96 589 261 268 261
query97 4460 4273 4244 4244
query98 216 203 200 200
query99 946 759 721 721
Total cold run time: 251176 ms
Total hot run time: 178564 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #31212:
URL: https://github.com/apache/doris/pull/31212#issuecomment-1956080980
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #31212:
URL: https://github.com/apache/doris/pull/31212
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Release StmtExecutor in AnalysisTask object when sql execution finished to release memory. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31212:
URL: https://github.com/apache/doris/pull/31212#issuecomment-1956121240
<details>
<summary>TPC-H: <b>Total hot run time: 41551 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit acc95a63fe6b7fc549e8c875d0afd5971aab9651, data reload: false
------ Round 1 ----------------------------------
q1 17634 5028 4942 4942
q2 2038 142 134 134
q3 10582 1029 1046 1029
q4 4655 977 991 977
q5 7698 3218 3278 3218
q6 199 139 140 139
q7 1266 782 779 779
q8 9250 2094 2100 2094
q9 7579 6724 6664 6664
q10 8313 2638 2625 2625
q11 419 235 216 216
q12 747 334 336 334
q13 18024 3713 3678 3678
q14 300 264 259 259
q15 628 553 546 546
q16 479 405 409 405
q17 927 863 839 839
q18 7513 6619 6522 6522
q19 1538 1489 1492 1489
q20 611 347 353 347
q21 6681 3966 3970 3966
q22 886 349 351 349
Total cold run time: 107967 ms
Total hot run time: 41551 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4884 4874 4859 4859
q2 299 193 195 193
q3 3651 3605 3633 3605
q4 2583 2592 2575 2575
q5 5784 5765 5749 5749
q6 213 131 134 131
q7 2277 1661 1657 1657
q8 3071 3153 3157 3153
q9 8749 8722 8721 8721
q10 6826 4264 4255 4255
q11 530 364 380 364
q12 778 541 555 541
q13 4347 3458 3454 3454
q14 280 233 229 229
q15 596 506 508 506
q16 491 458 471 458
q17 1710 1656 1633 1633
q18 8321 7740 7687 7687
q19 1642 1651 1630 1630
q20 2134 1835 1840 1835
q21 6817 6370 6267 6267
q22 613 534 533 533
Total cold run time: 66596 ms
Total hot run time: 60035 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org