You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hust-hhb (via GitHub)" <gi...@apache.org> on 2024/01/11 07:03:58 UTC
[PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
hust-hhb opened a new pull request, #29848:
URL: https://github.com/apache/doris/pull/29848
wal file on materialized-view will have repeat column id, when relaying it, need to consider deduplication.
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886425496
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886505098
<details>
<summary>ClickBench: <b>Total hot run time: 30.27 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 77368bc58ae230469b0407e97f07246ad1cdbd56, data reload: false
query1 0.05 0.06 0.05
query2 0.06 0.03 0.03
query3 0.25 0.11 0.10
query4 1.77 0.13 0.12
query5 0.52 0.52 0.52
query6 1.37 0.64 0.66
query7 0.02 0.01 0.01
query8 0.05 0.02 0.02
query9 0.54 0.50 0.50
query10 0.56 0.54 0.53
query11 0.13 0.09 0.09
query12 0.12 0.10 0.09
query13 0.60 0.60 0.60
query14 0.80 0.78 0.80
query15 0.81 0.81 0.82
query16 0.35 0.36 0.35
query17 0.98 0.96 0.98
query18 0.26 0.24 0.24
query19 1.86 1.76 1.77
query20 0.02 0.01 0.01
query21 15.40 0.57 0.57
query22 2.49 2.02 1.53
query23 17.17 1.07 0.83
query24 16.18 0.61 0.58
query25 2.16 0.15 0.15
query26 0.14 0.14 0.14
query27 0.14 0.15 0.15
query28 5.85 0.81 0.84
query29 12.55 3.15 3.27
query30 0.52 0.48 0.48
query31 2.78 0.36 0.37
query32 3.35 0.49 0.48
query33 3.19 3.22 3.23
query34 15.83 4.16 4.14
query35 4.18 4.18 4.16
query36 1.10 1.05 1.06
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.02
query40 0.16 0.13 0.13
query41 0.07 0.01 0.02
query42 0.03 0.02 0.01
query43 0.02 0.02 0.01
Total cold run time: 114.56 s
Total hot run time: 30.27 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
Posted by "hust-hhb (via GitHub)" <gi...@apache.org>.
hust-hhb commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886419908
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886534165
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.47 seconds
stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.4 seconds inserted 10000000 Rows, about 467K ops/s
storage size: 17183782566 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886428110
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886440530
<details>
<summary>TPC-H: <b>Total hot run time: 38989 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 77368bc58ae230469b0407e97f07246ad1cdbd56, data reload: false
------ Round 1 ----------------------------------
q1 17896 5044 4992 4992
q2 2041 148 136 136
q3 10830 1166 1206 1166
q4 10614 839 779 779
q5 8617 3246 3247 3246
q6 209 128 125 125
q7 905 524 516 516
q8 9586 2026 2024 2024
q9 7946 6636 6606 6606
q10 8367 3066 3052 3052
q11 418 202 201 201
q12 359 189 195 189
q13 18129 3448 3449 3448
q14 245 214 212 212
q15 556 520 501 501
q16 429 373 377 373
q17 948 497 552 497
q18 7373 6799 6701 6701
q19 1596 1366 1291 1291
q20 532 313 294 294
q21 2807 2355 2417 2355
q22 347 298 285 285
Total cold run time: 110750 ms
Total hot run time: 38989 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4963 4951 4935 4935
q2 313 205 203 203
q3 3331 3303 3301 3301
q4 2239 2245 2245 2245
q5 5822 5790 5788 5788
q6 191 118 116 116
q7 2318 1860 1883 1860
q8 3448 3576 3590 3576
q9 8861 8875 8821 8821
q10 3782 3771 3834 3771
q11 563 441 418 418
q12 788 608 605 605
q13 3882 3234 3205 3205
q14 289 267 271 267
q15 563 519 513 513
q16 503 473 465 465
q17 2061 2035 2046 2035
q18 8713 8333 8269 8269
q19 1594 1624 1611 1611
q20 2173 1932 1960 1932
q21 6116 5643 5666 5643
q22 540 456 473 456
Total cold run time: 63053 ms
Total hot run time: 60035 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886799451
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886481894
<details>
<summary>TPC-DS: <b>Total hot run time: 178327 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 77368bc58ae230469b0407e97f07246ad1cdbd56, data reload: false
query1 954 339 336 336
query2 6683 1965 2144 1965
query3 6715 219 212 212
query4 26099 22319 22032 22032
query5 5874 517 493 493
query6 267 189 185 185
query7 4613 278 281 278
query8 233 210 194 194
query9 9128 2774 2777 2774
query10 582 252 270 252
query11 16118 15347 15320 15320
query12 136 89 79 79
query13 1615 347 347 347
query14 11997 7393 7083 7083
query15 251 192 188 188
query16 6433 284 270 270
query17 1915 494 497 494
query18 1956 280 270 270
query19 294 144 137 137
query20 87 78 80 78
query21 186 91 95 91
query22 4971 4583 4803 4583
query23 31659 30899 30946 30899
query24 11990 2840 2807 2807
query25 567 349 343 343
query26 1703 156 151 151
query27 2841 274 285 274
query28 7148 1875 1869 1869
query29 1715 398 385 385
query30 295 149 155 149
query31 962 800 781 781
query32 99 65 58 58
query33 749 263 267 263
query34 901 450 444 444
query35 864 734 740 734
query36 1339 1272 1298 1272
query37 104 70 71 70
query38 3407 3319 3288 3288
query39 1320 1289 1286 1286
query40 314 95 93 93
query41 40 37 36 36
query42 103 92 97 92
query43 556 534 526 526
query44 1140 723 732 723
query45 217 186 185 185
query46 1063 657 689 657
query47 1663 1583 1565 1565
query48 386 307 321 307
query49 1240 330 320 320
query50 740 330 328 328
query51 5252 5263 5218 5218
query52 95 92 96 92
query53 223 156 154 154
query54 1417 560 599 560
query55 95 81 91 81
query56 217 191 205 191
query57 1017 941 948 941
query58 238 217 211 211
query59 2765 2614 2621 2614
query60 267 233 264 233
query61 84 84 83 83
query62 620 416 406 406
query63 174 155 155 155
query64 5815 1734 1822 1734
query65 3335 3276 3262 3262
query66 1142 341 327 327
query67 15968 15346 15399 15346
query68 12131 537 551 537
query69 580 282 281 281
query70 1729 1574 1645 1574
query71 505 238 222 222
query72 4966 2851 2839 2839
query73 2422 332 330 330
query74 6987 6411 6456 6411
query75 5050 2266 2302 2266
query76 6312 1110 1114 1110
query77 877 289 270 270
query78 9853 8719 8569 8569
query79 1044 526 520 520
query80 661 365 350 350
query81 462 215 213 213
query82 211 91 87 87
query83 162 136 139 136
query84 268 65 57 57
query85 981 298 288 288
query86 427 386 421 386
query87 3574 3388 3384 3384
query88 3175 2238 2270 2238
query89 341 263 248 248
query90 1878 216 208 208
query91 170 135 148 135
query92 64 53 54 53
query93 1579 497 441 441
query94 805 192 187 187
query95 497 441 428 428
query96 638 323 319 319
query97 4320 4148 4170 4148
query98 210 197 187 187
query99 988 731 747 731
Total cold run time: 294823 ms
Total hot run time: 178327 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #29848:
URL: https://github.com/apache/doris/pull/29848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886514578
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 77368bc58ae230469b0407e97f07246ad1cdbd56 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 12.4 seconds inserted 10000000 Rows, about 806K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [bug](group_commit) fix relay wal problem on materialized-view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29848:
URL: https://github.com/apache/doris/pull/29848#issuecomment-1886446968
TeamCity be ut coverage result:
Function Coverage: 36.72% (8650/23558)
Line Coverage: 28.75% (70469/245123)
Region Coverage: 27.67% (36406/131584)
Branch Coverage: 24.38% (18629/76422)
Coverage Report: http://coverage.selectdb-in.cc/coverage/77368bc58ae230469b0407e97f07246ad1cdbd56_77368bc58ae230469b0407e97f07246ad1cdbd56/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org