You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hqx871 (via GitHub)" <gi...@apache.org> on 2024/06/16 05:49:13 UTC
[PR] [feat](agg) support second key [doris]
hqx871 opened a new pull request, #36352:
URL: https://github.com/apache/doris/pull/36352
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
For uniq and agg table, Sometimes use will compute md5 value of some key columns and put md5 as key while put those key as value. For example
- when key column is too many, using md5 as key will reduce readed key columns as agg/uniq table will need to read all the key columns although user do not select those key columns
- use may need to use array as key, but current doris does not support
To improve those cases, I add a table property 'second_key' to let ScanNode to push predicates of those key columns to rowset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171256472
<details>
<summary>TPC-DS: <b>Total hot run time: 173211 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit efaf0c4f484996fd691b4aff2c2508b0c382c823, data reload: false
query1 911 385 377 377
query2 6456 2509 2376 2376
query3 6628 204 212 204
query4 22261 17308 17254 17254
query5 3636 476 446 446
query6 248 158 149 149
query7 4647 297 300 297
query8 318 289 284 284
query9 8469 2368 2336 2336
query10 573 291 273 273
query11 10703 10043 10083 10043
query12 113 86 83 83
query13 1630 365 359 359
query14 9164 6710 6790 6710
query15 230 186 187 186
query16 7222 263 272 263
query17 1404 568 530 530
query18 1917 282 278 278
query19 197 163 159 159
query20 90 85 83 83
query21 218 131 123 123
query22 4394 3960 4032 3960
query23 33772 33675 33566 33566
query24 11394 2836 2864 2836
query25 624 372 367 367
query26 1283 162 164 162
query27 2390 327 326 326
query28 7215 2120 2107 2107
query29 898 654 610 610
query30 251 153 154 153
query31 993 775 779 775
query32 86 56 52 52
query33 753 288 304 288
query34 1031 502 472 472
query35 783 676 651 651
query36 1174 988 964 964
query37 151 84 73 73
query38 2994 2837 2805 2805
query39 921 779 817 779
query40 217 127 125 125
query41 54 49 52 49
query42 117 94 97 94
query43 566 553 538 538
query44 1225 713 732 713
query45 192 159 166 159
query46 1076 740 710 710
query47 1858 1782 1763 1763
query48 364 300 289 289
query49 854 405 392 392
query50 770 383 399 383
query51 6791 6719 6680 6680
query52 98 90 102 90
query53 357 284 295 284
query54 880 448 439 439
query55 75 75 71 71
query56 274 250 260 250
query57 1123 1046 1034 1034
query58 245 256 257 256
query59 3544 3340 3201 3201
query60 286 261 269 261
query61 94 94 103 94
query62 637 441 433 433
query63 311 283 290 283
query64 8897 2295 1775 1775
query65 3170 3151 3108 3108
query66 884 326 334 326
query67 15513 15064 14964 14964
query68 4564 530 534 530
query69 598 478 395 395
query70 1199 1152 1095 1095
query71 416 267 262 262
query72 8393 5359 6028 5359
query73 776 324 324 324
query74 5969 5559 5463 5463
query75 3386 2632 2618 2618
query76 2717 910 945 910
query77 646 308 300 300
query78 10475 9880 9829 9829
query79 2486 500 516 500
query80 979 469 482 469
query81 601 219 231 219
query82 1361 98 102 98
query83 263 166 160 160
query84 239 90 83 83
query85 1405 289 331 289
query86 454 320 325 320
query87 3258 3081 3125 3081
query88 4312 2354 2340 2340
query89 475 394 386 386
query90 1695 188 192 188
query91 124 97 94 94
query92 59 47 46 46
query93 2350 502 498 498
query94 1040 186 183 183
query95 394 312 306 306
query96 597 265 268 265
query97 3308 3052 3043 3043
query98 220 200 194 194
query99 1207 854 841 841
Total cold run time: 274580 ms
Total hot run time: 173211 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171064712
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171266367
<details>
<summary>ClickBench: <b>Total hot run time: 30.08 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit efaf0c4f484996fd691b4aff2c2508b0c382c823, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.03 0.04
query3 0.23 0.04 0.05
query4 1.67 0.08 0.08
query5 0.49 0.47 0.50
query6 1.13 0.72 0.72
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.55 0.49 0.49
query10 0.55 0.56 0.54
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.60 0.59 0.60
query14 0.78 0.78 0.77
query15 0.84 0.82 0.81
query16 0.34 0.37 0.37
query17 1.04 1.02 0.98
query18 0.22 0.26 0.23
query19 1.80 1.65 1.75
query20 0.02 0.02 0.01
query21 15.40 0.65 0.65
query22 4.00 7.70 1.46
query23 18.29 1.31 1.36
query24 2.10 0.23 0.22
query25 0.15 0.08 0.09
query26 0.27 0.17 0.18
query27 0.08 0.07 0.08
query28 13.17 1.02 0.99
query29 12.64 3.22 3.24
query30 0.26 0.07 0.07
query31 2.84 0.38 0.38
query32 3.26 0.48 0.47
query33 2.91 2.87 2.87
query34 17.04 4.54 4.49
query35 4.47 4.51 4.48
query36 0.66 0.46 0.47
query37 0.18 0.15 0.15
query38 0.15 0.15 0.14
query39 0.05 0.03 0.03
query40 0.17 0.15 0.15
query41 0.10 0.05 0.05
query42 0.06 0.05 0.05
query43 0.04 0.03 0.04
Total cold run time: 109.03 s
Total hot run time: 30.08 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171230655
<details>
<summary>TPC-H: <b>Total hot run time: 40774 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit efaf0c4f484996fd691b4aff2c2508b0c382c823, data reload: false
------ Round 1 ----------------------------------
q1 17600 4432 4302 4302
q2 2020 196 198 196
q3 10448 1191 1138 1138
q4 10200 811 766 766
q5 7481 2689 2665 2665
q6 221 138 140 138
q7 962 619 610 610
q8 9223 2106 2122 2106
q9 8856 6561 6551 6551
q10 8959 3731 3710 3710
q11 462 235 239 235
q12 486 234 229 229
q13 17979 3004 2988 2988
q14 257 223 220 220
q15 518 489 490 489
q16 515 373 373 373
q17 989 681 747 681
q18 8144 7517 7331 7331
q19 5938 1554 1575 1554
q20 663 298 316 298
q21 5023 3863 3872 3863
q22 394 332 331 331
Total cold run time: 117338 ms
Total hot run time: 40774 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4407 4279 4236 4236
q2 372 269 262 262
q3 2975 3031 2936 2936
q4 1966 1655 1768 1655
q5 5554 5515 5510 5510
q6 229 134 138 134
q7 2210 1895 1843 1843
q8 3335 3404 3523 3404
q9 8773 8732 8887 8732
q10 4070 3934 3768 3768
q11 579 498 512 498
q12 818 648 614 614
q13 17001 3171 3188 3171
q14 288 290 274 274
q15 537 495 476 476
q16 482 421 429 421
q17 1853 1542 1520 1520
q18 8095 7922 7794 7794
q19 1851 1658 1715 1658
q20 3124 1929 1842 1842
q21 5312 5095 4870 4870
q22 642 535 558 535
Total cold run time: 74473 ms
Total hot run time: 56153 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171062724
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171068594
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171266909
TeamCity be ut coverage result:
Function Coverage: 36.47% (9001/24678)
Line Coverage: 28.01% (73716/263133)
Region Coverage: 27.49% (38295/139319)
Branch Coverage: 24.18% (19520/80716)
Coverage Report: http://coverage.selectdb-in.cc/coverage/efaf0c4f484996fd691b4aff2c2508b0c382c823_efaf0c4f484996fd691b4aff2c2508b0c382c823/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #36352: [feat](agg) support second key
URL: https://github.com/apache/doris/pull/36352
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171683228
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2543384965
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171116046
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](agg) support second key [doris]
Posted by "hqx871 (via GitHub)" <gi...@apache.org>.
hqx871 commented on PR #36352:
URL: https://github.com/apache/doris/pull/36352#issuecomment-2171118735
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org