You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wuwenchi (via GitHub)" <gi...@apache.org> on 2024/03/07 07:03:44 UTC
[PR] [improvement](iceberg/paimon)support estimate row count for 2.0 [doris]
wuwenchi opened a new pull request, #31927:
URL: https://github.com/apache/doris/pull/31927
## Proposed changes
mirror: #31204 #31473
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg/paimon)support estimate row count for 2.0 [doris]
Posted by "wuwenchi (via GitHub)" <gi...@apache.org>.
wuwenchi commented on PR #31927:
URL: https://github.com/apache/doris/pull/31927#issuecomment-1982651468
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg/paimon)support estimate row count for 2.0 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31927:
URL: https://github.com/apache/doris/pull/31927#issuecomment-1982650551
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg/paimon)support estimate row count for 2.0 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31927:
URL: https://github.com/apache/doris/pull/31927#issuecomment-1982841621
<details>
<summary>TPC-H: <b>Total hot run time: 49920 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5b9c0283ad37d61fb6ac41d50a775b537c734374, data reload: false
------ Round 1 ----------------------------------
q1 17740 4407 4373 4373
q2 2029 148 139 139
q3 10329 1875 1905 1875
q4 10359 1247 1326 1247
q5 8383 3925 3960 3925
q6 225 124 120 120
q7 2022 1604 1609 1604
q8 9523 2688 2688 2688
q9 13217 10867 10698 10698
q10 8644 3530 3480 3480
q11 417 234 244 234
q12 457 290 301 290
q13 18315 3954 4029 3954
q14 352 328 334 328
q15 504 457 455 455
q16 696 586 588 586
q17 1107 939 902 902
q18 7178 6705 6897 6705
q19 1703 1566 1586 1566
q20 505 298 298 298
q21 4383 4046 4060 4046
q22 509 407 409 407
Total cold run time: 118597 ms
Total hot run time: 49920 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4330 4307 4244 4244
q2 311 216 214 214
q3 4142 4149 4106 4106
q4 2786 2734 2751 2734
q5 7303 7199 7214 7199
q6 234 118 115 115
q7 3255 2873 2824 2824
q8 4324 4413 4467 4413
q9 17739 17233 17244 17233
q10 4248 4268 4280 4268
q11 743 707 684 684
q12 1037 860 844 844
q13 6756 3765 3734 3734
q14 450 419 430 419
q15 498 464 455 455
q16 757 702 717 702
q17 3788 3907 3906 3906
q18 9271 9268 8930 8930
q19 1723 1653 1687 1653
q20 2394 2203 2178 2178
q21 8403 8455 8569 8455
q22 1041 982 926 926
Total cold run time: 85533 ms
Total hot run time: 80236 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg/paimon)support estimate row count for 2.0 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31927:
URL: https://github.com/apache/doris/pull/31927#issuecomment-1982895361
<details>
<summary>ClickBench: <b>Total hot run time: 29.74 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5b9c0283ad37d61fb6ac41d50a775b537c734374, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.02
query3 0.25 0.05 0.04
query4 1.83 0.07 0.05
query5 0.53 0.52 0.51
query6 1.25 0.62 0.62
query7 0.02 0.01 0.02
query8 0.03 0.03 0.03
query9 0.53 0.48 0.48
query10 0.54 0.55 0.54
query11 0.12 0.09 0.09
query12 0.11 0.09 0.09
query13 0.62 0.62 0.62
query14 0.78 0.79 0.79
query15 0.77 0.75 0.77
query16 0.37 0.35 0.37
query17 1.02 1.01 1.02
query18 0.22 0.24 0.25
query19 1.90 1.82 1.77
query20 0.02 0.01 0.01
query21 15.48 0.56 0.56
query22 2.29 3.14 0.91
query23 17.24 0.93 0.82
query24 4.78 1.71 0.73
query25 0.35 0.13 0.06
query26 0.58 0.15 0.16
query27 0.03 0.04 0.04
query28 8.46 0.75 0.70
query29 12.76 2.35 2.39
query30 0.62 0.54 0.54
query31 2.81 0.39 0.37
query32 3.41 0.51 0.50
query33 3.03 3.07 3.09
query34 15.30 4.81 4.80
query35 4.86 4.84 4.86
query36 1.07 1.01 1.03
query37 0.06 0.04 0.04
query38 0.03 0.02 0.02
query39 0.02 0.01 0.02
query40 0.15 0.14 0.14
query41 0.06 0.01 0.02
query42 0.02 0.01 0.01
query43 0.02 0.02 0.02
Total cold run time: 104.43 s
Total hot run time: 29.74 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg/paimon)support estimate row count for 2.0 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31927:
URL: https://github.com/apache/doris/pull/31927#issuecomment-1982904909
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 5b9c0283ad37d61fb6ac41d50a775b537c734374 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 20.7 seconds inserted 10000000 Rows, about 483K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg/paimon)support estimate row count for 2.0 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31927:
URL: https://github.com/apache/doris/pull/31927#issuecomment-1982879108
<details>
<summary>TPC-DS: <b>Total hot run time: 201628 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5b9c0283ad37d61fb6ac41d50a775b537c734374, data reload: false
query1 927 388 381 381
query2 6518 2198 2164 2164
query3 6919 200 198 198
query4 20798 17939 17781 17781
query5 19731 6492 6504 6492
query6 317 214 234 214
query7 4381 295 287 287
query8 249 216 232 216
query9 3104 2689 2617 2617
query10 422 288 303 288
query11 11292 10618 10710 10618
query12 125 73 69 69
query13 5589 651 634 634
query14 18361 13424 13476 13424
query15 360 226 238 226
query16 6444 268 268 268
query17 1572 1445 872 872
query18 2308 404 395 395
query19 197 140 147 140
query20 77 77 75 75
query21 183 92 97 92
query22 5153 5139 5109 5109
query23 32401 31990 31810 31810
query24 6897 6494 6525 6494
query25 508 414 404 404
query26 511 163 157 157
query27 1786 287 291 287
query28 6089 2246 2202 2202
query29 2786 2677 2647 2647
query30 234 157 161 157
query31 910 707 735 707
query32 64 60 56 56
query33 391 251 248 248
query34 852 455 465 455
query35 1105 892 880 880
query36 1733 1414 1564 1414
query37 90 62 60 60
query38 3049 2873 2937 2873
query39 1393 1302 1304 1302
query40 204 84 89 84
query41 36 32 31 31
query42 91 84 88 84
query43 653 645 594 594
query44 1116 726 715 715
query45 240 226 226 226
query46 1228 977 950 950
query47 1811 1719 1881 1719
query48 975 695 662 662
query49 613 369 364 364
query50 853 596 601 596
query51 4712 4682 4648 4648
query52 80 76 72 72
query53 446 310 316 310
query54 2645 2460 2527 2460
query55 87 68 78 68
query56 201 184 191 184
query57 1174 1192 1104 1104
query58 212 198 199 198
query59 3743 3192 3427 3192
query60 210 197 186 186
query61 87 84 88 84
query62 798 536 473 473
query63 492 337 330 330
query64 2234 1495 1461 1461
query65 3591 3576 3566 3566
query66 763 374 364 364
query67 17109 15403 16157 15403
query68 8223 649 656 649
query69 556 353 345 345
query70 2006 1784 1512 1512
query71 409 305 309 305
query72 6444 3444 3412 3412
query73 722 321 324 321
query74 6258 5814 5881 5814
query75 4677 3664 3717 3664
query76 4617 1146 1202 1146
query77 536 249 255 249
query78 13020 11685 12691 11685
query79 9690 632 627 627
query80 2532 392 387 387
query81 534 229 236 229
query82 1422 95 92 92
query83 178 129 135 129
query84 255 68 67 67
query85 1359 286 280 280
query86 405 389 416 389
query87 3252 3007 3014 3007
query88 5008 2291 2296 2291
query89 387 308 271 271
query90 1757 195 195 195
query91 149 115 115 115
query92 55 49 52 49
query93 5417 580 558 558
query94 908 202 193 193
query95 1092 1051 1062 1051
query96 624 317 308 308
query97 6427 6375 6445 6375
query98 189 178 190 178
query99 2715 905 894 894
Total cold run time: 314053 ms
Total hot run time: 201628 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](iceberg/paimon)support estimate row count for 2.0 [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #31927:
URL: https://github.com/apache/doris/pull/31927
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org