You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zddr (via GitHub)" <gi...@apache.org> on 2024/01/19 12:13:44 UTC
[PR] [fix](mtmv)return MTMV with at least one available partition [doris]
zddr opened a new pull request, #30156:
URL: https://github.com/apache/doris/pull/30156
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)return MTMV with at least one available partition [doris]
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on code in PR #30156:
URL: https://github.com/apache/doris/pull/30156#discussion_r1458921482
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java:
##########
@@ -70,7 +70,7 @@ private void initMaterializationContext(CascadesContext cascadesContext) {
List<BaseTableInfo> usedBaseTables =
collectedTables.stream().map(BaseTableInfo::new).collect(Collectors.toList());
Set<MTMV> availableMTMVs = Env.getCurrentEnv().getMtmvService().getRelationManager()
- .getAvailableMTMVs(usedBaseTables);
+ .getAvailableMTMVs(usedBaseTables, cascadesContext.getConnectContext());
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)return MTMV with at least one available partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30156:
URL: https://github.com/apache/doris/pull/30156#issuecomment-1900394308
<details>
<summary>ClickBench: <b>Total hot run time: 31.35 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ecac40f2ff3255361275833140e708acfb2a668e, data reload: false
query1 0.04 0.03 0.03
query2 0.05 0.03 0.02
query3 0.22 0.04 0.05
query4 1.70 0.07 0.06
query5 0.54 0.52 0.52
query6 1.37 0.63 0.64
query7 0.02 0.01 0.01
query8 0.04 0.02 0.03
query9 0.55 0.50 0.50
query10 0.57 0.57 0.56
query11 0.12 0.09 0.09
query12 0.11 0.09 0.09
query13 0.60 0.62 0.61
query14 0.77 0.80 0.82
query15 0.79 0.77 0.78
query16 0.37 0.37 0.40
query17 1.01 1.02 0.99
query18 0.22 0.25 0.25
query19 1.89 1.78 1.78
query20 0.01 0.01 0.02
query21 15.40 0.59 0.56
query22 2.43 2.62 2.02
query23 17.54 0.83 0.79
query24 2.65 1.12 1.13
query25 0.41 0.26 0.18
query26 0.54 0.13 0.14
query27 0.06 0.05 0.04
query28 11.34 0.76 0.78
query29 12.50 3.21 3.18
query30 0.54 0.50 0.48
query31 2.78 0.34 0.34
query32 3.39 0.49 0.48
query33 3.22 3.24 3.22
query34 15.85 4.28 4.28
query35 4.40 4.28 4.28
query36 1.13 1.08 1.08
query37 0.06 0.05 0.05
query38 0.03 0.02 0.03
query39 0.02 0.02 0.02
query40 0.16 0.13 0.15
query41 0.06 0.01 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 105.54 s
Total hot run time: 31.35 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)return MTMV with at least one available partition [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on code in PR #30156:
URL: https://github.com/apache/doris/pull/30156#discussion_r1458915572
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java:
##########
@@ -70,7 +70,7 @@ private void initMaterializationContext(CascadesContext cascadesContext) {
List<BaseTableInfo> usedBaseTables =
collectedTables.stream().map(BaseTableInfo::new).collect(Collectors.toList());
Set<MTMV> availableMTMVs = Env.getCurrentEnv().getMtmvService().getRelationManager()
- .getAvailableMTMVs(usedBaseTables);
+ .getAvailableMTMVs(usedBaseTables, cascadesContext.getConnectContext());
Review Comment:
should make sure that this connect connext should not be modified later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)return MTMV with at least one available partition [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on code in PR #30156:
URL: https://github.com/apache/doris/pull/30156#discussion_r1458915572
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java:
##########
@@ -70,7 +70,7 @@ private void initMaterializationContext(CascadesContext cascadesContext) {
List<BaseTableInfo> usedBaseTables =
collectedTables.stream().map(BaseTableInfo::new).collect(Collectors.toList());
Set<MTMV> availableMTMVs = Env.getCurrentEnv().getMtmvService().getRelationManager()
- .getAvailableMTMVs(usedBaseTables);
+ .getAvailableMTMVs(usedBaseTables, cascadesContext.getConnectContext());
Review Comment:
should make sure that this sessionVarible of connect connext should not be modified later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)return MTMV with at least one available partition [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #30156:
URL: https://github.com/apache/doris/pull/30156
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)return MTMV with at least one available partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30156:
URL: https://github.com/apache/doris/pull/30156#issuecomment-1900353685
<details>
<summary>TPC-H: <b>Total hot run time: 38826 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ecac40f2ff3255361275833140e708acfb2a668e, data reload: false
------ Round 1 ----------------------------------
q1 17705 5230 5340 5230
q2 2026 139 135 135
q3 10638 1133 1143 1133
q4 10226 771 844 771
q5 7776 3137 3133 3133
q6 197 123 125 123
q7 871 490 480 480
q8 9268 1936 1942 1936
q9 7231 6373 6355 6355
q10 8194 3086 3028 3028
q11 430 213 209 209
q12 352 188 187 187
q13 17991 3352 3310 3310
q14 240 208 217 208
q15 551 502 505 502
q16 428 376 364 364
q17 938 496 498 496
q18 7323 6854 6872 6854
q19 1605 1450 1369 1369
q20 555 292 307 292
q21 2778 2396 2447 2396
q22 373 315 316 315
Total cold run time: 107696 ms
Total hot run time: 38826 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5302 5316 5195 5195
q2 322 220 209 209
q3 3306 3203 3249 3203
q4 2047 2076 2054 2054
q5 5993 5957 5803 5803
q6 198 116 122 116
q7 2319 1861 1873 1861
q8 3198 3359 3378 3359
q9 8907 8752 8732 8732
q10 3890 3846 3873 3846
q11 547 442 448 442
q12 797 648 620 620
q13 16968 3166 3111 3111
q14 282 256 264 256
q15 543 503 502 502
q16 522 447 466 447
q17 1920 1830 1856 1830
q18 9366 17504 9634 9634
q19 25100 1575 1557 1557
q20 4587 1938 1915 1915
q21 14419 5486 5472 5472
q22 1013 569 546 546
Total cold run time: 111546 ms
Total hot run time: 60710 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)return MTMV with at least one available partition [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30156:
URL: https://github.com/apache/doris/pull/30156#issuecomment-1900317234
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)return MTMV with at least one available partition [doris]
Posted by "zddr (via GitHub)" <gi...@apache.org>.
zddr commented on PR #30156:
URL: https://github.com/apache/doris/pull/30156#issuecomment-1900311023
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)return MTMV with at least one available partition [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30156:
URL: https://github.com/apache/doris/pull/30156#issuecomment-1903156681
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)return MTMV with at least one available partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30156:
URL: https://github.com/apache/doris/pull/30156#issuecomment-1900398558
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit ecac40f2ff3255361275833140e708acfb2a668e with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.8 seconds inserted 10000000 Rows, about 675K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv)return MTMV with at least one available partition [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30156:
URL: https://github.com/apache/doris/pull/30156#issuecomment-1900383435
<details>
<summary>TPC-DS: <b>Total hot run time: 177342 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ecac40f2ff3255361275833140e708acfb2a668e, data reload: false
query1 933 333 328 328
query2 6554 1999 1937 1937
query3 6714 199 201 199
query4 33767 22128 22128 22128
query5 6905 518 542 518
query6 250 175 156 156
query7 4604 262 255 255
query8 228 178 174 174
query9 8344 2580 2538 2538
query10 448 231 248 231
query11 16946 15766 15486 15486
query12 120 66 67 66
query13 1695 392 390 390
query14 10495 7206 7009 7009
query15 220 178 191 178
query16 4605 232 231 231
query17 992 493 485 485
query18 1794 262 260 260
query19 176 137 138 137
query20 78 73 67 67
query21 183 135 127 127
query22 5145 4850 4915 4850
query23 31441 30922 30969 30922
query24 12529 2832 2836 2832
query25 573 319 323 319
query26 1636 138 148 138
query27 3213 274 283 274
query28 7121 1814 1800 1800
query29 1652 646 644 644
query30 280 134 140 134
query31 933 750 766 750
query32 73 50 53 50
query33 704 221 214 214
query34 1145 467 481 467
query35 916 783 771 771
query36 1372 1224 1233 1224
query37 90 58 60 58
query38 3333 3267 3205 3205
query39 1318 1270 1264 1264
query40 341 94 86 86
query41 38 37 39 37
query42 91 80 84 80
query43 501 481 491 481
query44 1094 686 688 686
query45 189 184 172 172
query46 1078 654 646 646
query47 1702 1624 1638 1624
query48 394 316 309 309
query49 1168 286 294 286
query50 681 313 315 313
query51 5291 5310 5260 5260
query52 95 77 87 77
query53 326 256 265 256
query54 475 294 286 286
query55 82 77 80 77
query56 181 175 169 169
query57 989 980 986 980
query58 189 161 159 159
query59 2931 2774 2754 2754
query60 220 186 182 182
query61 86 88 83 83
query62 612 374 359 359
query63 277 259 261 259
query64 6133 1768 1758 1758
query65 3342 3269 3256 3256
query66 1226 325 315 315
query67 15868 15410 14912 14912
query68 11608 527 485 485
query69 629 312 302 302
query70 1706 1491 1423 1423
query71 554 211 211 211
query72 4951 2860 2848 2848
query73 2066 313 316 313
query74 7061 6350 6378 6350
query75 4861 2320 2310 2310
query76 6702 1072 1024 1024
query77 657 237 243 237
query78 9063 8986 8651 8651
query79 997 491 504 491
query80 546 319 311 311
query81 444 204 199 199
query82 199 81 78 78
query83 139 127 122 122
query84 277 72 68 68
query85 1065 333 329 329
query86 397 376 393 376
query87 3524 3403 3315 3315
query88 3081 2208 2190 2190
query89 444 390 355 355
query90 1979 191 188 188
query91 162 122 132 122
query92 52 45 43 43
query93 1097 443 429 429
query94 1148 165 160 160
query95 511 452 441 441
query96 603 316 315 315
query97 4250 4125 4151 4125
query98 201 200 180 180
query99 1025 673 683 673
Total cold run time: 297124 ms
Total hot run time: 177342 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org