You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mymeiyi (via GitHub)" <gi...@apache.org> on 2024/03/12 10:41:46 UTC
[PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
mymeiyi opened a new pull request, #32135:
URL: https://github.com/apache/doris/pull/32135
## Proposed changes
Issue Number: close #xxx
When use `group commmit` and `chunked stream load`, because `chunked stream load` does not has content length, we can estimate the wal size, so we skip group commit in the original way.
This pr allow it.
How to avoid large wal will be solved in later pr.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #32135:
URL: https://github.com/apache/doris/pull/32135#issuecomment-1991424661
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32135:
URL: https://github.com/apache/doris/pull/32135#issuecomment-1994721539
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32135:
URL: https://github.com/apache/doris/pull/32135#issuecomment-1994721671
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32135:
URL: https://github.com/apache/doris/pull/32135#issuecomment-1991536043
<details>
<summary>TPC-DS: <b>Total hot run time: 186755 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3bd3fbbef0cc708a3446e75e6e3b345a75ef5869, data reload: false
query1 906 356 347 347
query2 7422 2140 2023 2023
query3 6712 225 212 212
query4 31201 20612 20647 20612
query5 4338 400 420 400
query6 257 176 186 176
query7 4629 290 281 281
query8 229 174 173 173
query9 9881 2245 2214 2214
query10 455 235 262 235
query11 14542 14165 14253 14165
query12 130 85 79 79
query13 1615 407 402 402
query14 11178 10842 10360 10360
query15 271 221 189 189
query16 8106 270 272 270
query17 1934 568 540 540
query18 2105 285 278 278
query19 294 162 153 153
query20 92 86 88 86
query21 197 132 127 127
query22 4696 4390 4439 4390
query23 31809 31225 31099 31099
query24 10744 2823 2790 2790
query25 611 377 382 377
query26 1274 158 167 158
query27 2429 360 355 355
query28 7626 1855 1865 1855
query29 871 643 632 632
query30 301 146 146 146
query31 958 734 723 723
query32 96 61 58 58
query33 764 281 266 266
query34 1028 472 501 472
query35 831 619 607 607
query36 964 880 851 851
query37 122 77 78 77
query38 3326 3113 3201 3113
query39 1428 1377 1386 1377
query40 217 114 111 111
query41 51 48 46 46
query42 103 95 101 95
query43 497 467 467 467
query44 1118 730 697 697
query45 272 243 271 243
query46 1092 702 685 685
query47 1692 1608 1597 1597
query48 447 372 352 352
query49 1141 333 337 333
query50 762 383 384 383
query51 6635 6646 6486 6486
query52 109 91 94 91
query53 341 280 318 280
query54 307 250 242 242
query55 83 81 76 76
query56 261 239 225 225
query57 1077 1027 1007 1007
query58 240 199 205 199
query59 2743 2693 2671 2671
query60 267 273 254 254
query61 108 108 107 107
query62 600 408 398 398
query63 307 285 282 282
query64 5743 3765 3885 3765
query65 3101 3045 3020 3020
query66 815 327 327 327
query67 15040 14543 14800 14543
query68 6770 518 521 518
query69 619 378 377 377
query70 1289 1134 1172 1134
query71 486 270 263 263
query72 6509 2650 2479 2479
query73 733 322 319 319
query74 6989 6656 6533 6533
query75 9432 7980 7999 7980
query76 4356 839 904 839
query77 605 262 256 256
query78 10359 9634 9622 9622
query79 8699 529 527 527
query80 1635 376 373 373
query81 521 208 209 208
query82 879 204 202 202
query83 220 146 145 145
query84 281 75 75 75
query85 1471 322 310 310
query86 472 295 303 295
query87 3542 3231 3305 3231
query88 4971 2277 2271 2271
query89 500 352 363 352
query90 2001 181 173 173
query91 159 132 128 128
query92 63 45 46 45
query93 6664 501 496 496
query94 1112 178 173 173
query95 424 319 328 319
query96 614 278 270 270
query97 4440 4240 4321 4240
query98 226 216 207 207
query99 1117 762 745 745
Total cold run time: 309699 ms
Total hot run time: 186755 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #32135:
URL: https://github.com/apache/doris/pull/32135
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32135:
URL: https://github.com/apache/doris/pull/32135#issuecomment-1991500647
TeamCity be ut coverage result:
Function Coverage: 34.95% (8553/24470)
Line Coverage: 26.79% (69506/259419)
Region Coverage: 26.04% (36064/138507)
Branch Coverage: 23.01% (18438/80128)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3bd3fbbef0cc708a3446e75e6e3b345a75ef5869_3bd3fbbef0cc708a3446e75e6e3b345a75ef5869/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #32135:
URL: https://github.com/apache/doris/pull/32135#issuecomment-1991461360
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32135:
URL: https://github.com/apache/doris/pull/32135#issuecomment-1991514704
<details>
<summary>TPC-H: <b>Total hot run time: 36651 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3bd3fbbef0cc708a3446e75e6e3b345a75ef5869, data reload: false
------ Round 1 ----------------------------------
q1 17675 4142 4072 4072
q2 2021 154 142 142
q3 10568 905 897 897
q4 4650 675 737 675
q5 7595 2800 2826 2800
q6 184 125 126 125
q7 1221 822 808 808
q8 9476 2061 2015 2015
q9 7305 6407 6438 6407
q10 8195 2523 2456 2456
q11 425 229 224 224
q12 763 309 297 297
q13 17952 3034 3012 3012
q14 278 262 249 249
q15 496 458 451 451
q16 485 405 396 396
q17 960 516 561 516
q18 6636 5873 5778 5778
q19 1549 1451 1419 1419
q20 554 286 272 272
q21 7494 3333 3354 3333
q22 793 307 339 307
Total cold run time: 107275 ms
Total hot run time: 36651 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4086 4097 4050 4050
q2 319 225 220 220
q3 2922 2816 2833 2816
q4 1824 1575 1563 1563
q5 5204 5278 5263 5263
q6 195 116 124 116
q7 2239 1831 1815 1815
q8 3146 3323 3286 3286
q9 8518 8499 8511 8499
q10 6158 3686 3678 3678
q11 533 460 459 459
q12 721 536 554 536
q13 15648 2864 2830 2830
q14 264 257 271 257
q15 502 444 444 444
q16 469 401 412 401
q17 1718 1480 1443 1443
q18 7554 7230 7118 7118
q19 6163 1539 1555 1539
q20 1911 1736 1725 1725
q21 4872 4553 4619 4553
q22 507 456 448 448
Total cold run time: 75473 ms
Total hot run time: 53059 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32135:
URL: https://github.com/apache/doris/pull/32135#issuecomment-1991337178
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32135:
URL: https://github.com/apache/doris/pull/32135#issuecomment-1991545565
<details>
<summary>ClickBench: <b>Total hot run time: 29.57 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3bd3fbbef0cc708a3446e75e6e3b345a75ef5869, data reload: false
query1 0.04 0.04 0.04
query2 0.06 0.03 0.02
query3 0.22 0.06 0.06
query4 1.67 0.10 0.09
query5 0.50 0.48 0.52
query6 1.18 0.63 0.63
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.55 0.50 0.52
query10 0.55 0.54 0.56
query11 0.12 0.09 0.09
query12 0.11 0.09 0.09
query13 0.57 0.58 0.57
query14 0.74 0.74 0.75
query15 0.79 0.78 0.79
query16 0.41 0.39 0.39
query17 0.99 0.97 0.97
query18 0.19 0.27 0.21
query19 1.74 1.64 1.66
query20 0.01 0.01 0.01
query21 15.42 0.60 0.58
query22 2.65 2.58 1.27
query23 17.16 0.92 0.71
query24 4.32 1.59 0.72
query25 0.08 0.13 0.04
query26 0.85 0.12 0.11
query27 0.03 0.03 0.05
query28 10.35 0.92 0.87
query29 12.52 3.36 3.45
query30 0.67 0.63 0.56
query31 2.79 0.32 0.33
query32 3.41 0.44 0.44
query33 2.78 2.77 2.78
query34 16.10 4.31 4.31
query35 4.29 4.35 4.29
query36 1.08 1.00 1.00
query37 0.07 0.05 0.06
query38 0.05 0.03 0.03
query39 0.03 0.02 0.02
query40 0.17 0.15 0.14
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.03 0.03 0.02
Total cold run time: 105.46 s
Total hot run time: 29.57 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](group commit) Group commit support chunked stream load in flink [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32135:
URL: https://github.com/apache/doris/pull/32135#issuecomment-1991550973
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 3bd3fbbef0cc708a3446e75e6e3b345a75ef5869 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 17.4 seconds inserted 10000000 Rows, about 574K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org