You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Yukang-Lian (via GitHub)" <gi...@apache.org> on 2024/07/10 07:47:19 UTC
[PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]
Yukang-Lian opened a new pull request, #37598:
URL: https://github.com/apache/doris/pull/37598
Problem: An error is encountered when executing a schema change on a unique table to add a column with a complex type, such as bitmap, as documented in https://github.com/apache/doris/issues/31365
Reason: The error arises because the schema change logic erroneously applies an aggregation check for new columns across all table types, demanding an explicit aggregation type declaration. However, unique and duplicate tables inherently assume default aggregation types for newly added columns, leading to this misstep.
Solution: The schema change process for introducing new columns needs to distinguish between table types accurately. For unique and duplicate tables, it should automatically assign the appropriate aggregation type, which, for the purpose of smooth integration with subsequent processes, should be set to NONE.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219854578
<details>
<summary>TPC-DS: <b>Total hot run time: 203535 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf54ec39e051c2b896d658290bbe1e0e5bb4bdae, data reload: false
query1 935 434 379 379
query2 6529 2825 2509 2509
query3 6921 207 200 200
query4 21372 18132 18183 18132
query5 19745 6522 6483 6483
query6 305 220 226 220
query7 4164 300 309 300
query8 427 458 465 458
query9 3123 2649 2555 2555
query10 418 324 299 299
query11 11355 10702 10674 10674
query12 129 77 77 77
query13 5606 699 695 695
query14 17812 13745 13300 13300
query15 362 253 252 252
query16 6466 295 266 266
query17 1698 1439 899 899
query18 2286 419 414 414
query19 202 155 148 148
query20 84 80 80 80
query21 193 99 98 98
query22 5125 4988 4987 4987
query23 32778 31985 32020 31985
query24 6967 6500 6489 6489
query25 529 441 434 434
query26 525 170 167 167
query27 1874 302 303 302
query28 6194 2367 2281 2281
query29 3002 2789 2608 2608
query30 244 165 170 165
query31 927 760 767 760
query32 72 67 62 62
query33 402 257 267 257
query34 863 482 490 482
query35 1116 949 905 905
query36 1405 1180 1057 1057
query37 89 63 61 61
query38 3057 2947 2894 2894
query39 1392 1333 1347 1333
query40 207 100 96 96
query41 48 44 44 44
query42 85 82 85 82
query43 773 791 674 674
query44 1129 732 728 728
query45 246 238 236 236
query46 1231 978 953 953
query47 1774 1790 1789 1789
query48 1010 731 723 723
query49 620 387 369 369
query50 853 599 605 599
query51 4764 4661 4673 4661
query52 100 90 86 86
query53 458 324 340 324
query54 2664 2470 2475 2470
query55 86 81 87 81
query56 246 210 207 207
query57 1167 1143 1162 1143
query58 229 204 199 199
query59 4419 3674 4061 3674
query60 225 202 219 202
query61 111 93 95 93
query62 850 486 439 439
query63 490 355 348 348
query64 2454 1512 1535 1512
query65 3652 3563 3609 3563
query66 788 373 378 373
query67 17543 15315 15481 15315
query68 8439 659 660 659
query69 576 347 358 347
query70 1524 1428 1325 1325
query71 393 316 305 305
query72 6496 3496 3487 3487
query73 731 322 322 322
query74 6275 5918 5897 5897
query75 4774 3816 3672 3672
query76 4958 1180 1199 1180
query77 718 256 248 248
query78 12512 11682 11870 11682
query79 9354 651 667 651
query80 1633 419 407 407
query81 505 238 233 233
query82 1685 104 104 104
query83 166 133 135 133
query84 265 73 73 73
query85 892 332 324 324
query86 343 290 330 290
query87 3242 3018 3039 3018
query88 5178 2323 2332 2323
query89 419 302 351 302
query90 1927 224 216 216
query91 177 140 141 140
query92 61 56 55 55
query93 6271 579 580 579
query94 741 215 212 212
query95 1133 1064 1071 1064
query96 657 331 328 328
query97 6509 6296 6443 6296
query98 205 192 175 175
query99 2970 838 885 838
Total cold run time: 316644 ms
Total hot run time: 203535 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219877864
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit bf54ec39e051c2b896d658290bbe1e0e5bb4bdae with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 22.0 seconds inserted 10000000 Rows, about 454K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219869590
<details>
<summary>ClickBench: <b>Total hot run time: 30.88 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf54ec39e051c2b896d658290bbe1e0e5bb4bdae, data reload: false
query1 0.02 0.02 0.02
query2 0.06 0.03 0.02
query3 0.25 0.06 0.05
query4 1.76 0.07 0.07
query5 0.54 0.53 0.51
query6 1.31 0.65 0.62
query7 0.02 0.01 0.02
query8 0.03 0.03 0.02
query9 0.52 0.47 0.48
query10 0.55 0.54 0.54
query11 0.12 0.08 0.09
query12 0.12 0.09 0.09
query13 0.63 0.61 0.61
query14 0.80 0.79 0.80
query15 0.79 0.77 0.76
query16 0.37 0.35 0.35
query17 1.01 1.03 1.01
query18 0.24 0.24 0.24
query19 1.94 1.86 1.79
query20 0.02 0.01 0.01
query21 15.47 0.56 0.55
query22 1.96 1.99 1.66
query23 17.47 1.10 0.84
query24 6.55 1.20 1.33
query25 0.38 0.11 0.06
query26 0.77 0.16 0.15
query27 0.05 0.04 0.03
query28 6.07 0.84 0.73
query29 12.61 2.32 2.29
query30 0.57 0.54 0.53
query31 2.81 0.40 0.38
query32 3.38 0.51 0.50
query33 3.11 3.10 3.06
query34 15.28 4.77 4.80
query35 4.86 4.83 4.84
query36 1.08 1.00 1.01
query37 0.06 0.05 0.05
query38 0.03 0.02 0.02
query39 0.02 0.02 0.01
query40 0.16 0.14 0.13
query41 0.06 0.01 0.01
query42 0.03 0.01 0.01
query43 0.02 0.01 0.02
Total cold run time: 103.9 s
Total hot run time: 30.88 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219793252
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219828946
<details>
<summary>TPC-H: <b>Total hot run time: 50160 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf54ec39e051c2b896d658290bbe1e0e5bb4bdae, data reload: false
------ Round 1 ----------------------------------
q1 20259 4523 4568 4523
q2 2259 227 160 160
q3 12786 1967 1957 1957
q4 10111 1249 1327 1249
q5 8372 3851 3918 3851
q6 233 126 129 126
q7 2052 1636 1599 1599
q8 9476 2754 2729 2729
q9 10848 10262 10262 10262
q10 8673 3526 3543 3526
q11 408 247 256 247
q12 476 305 309 305
q13 18341 3941 4061 3941
q14 366 331 342 331
q15 524 455 456 455
q16 679 577 575 575
q17 1160 983 971 971
q18 7322 7058 6934 6934
q19 1795 1707 1649 1649
q20 548 292 308 292
q21 4432 4142 4031 4031
q22 537 462 447 447
Total cold run time: 121657 ms
Total hot run time: 50160 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4372 4302 4295 4295
q2 329 234 229 229
q3 4181 4136 4113 4113
q4 2757 2743 2759 2743
q5 7153 7115 7053 7053
q6 238 128 123 123
q7 3268 2809 2828 2809
q8 4424 4516 4488 4488
q9 16926 16781 16829 16781
q10 4266 4248 4287 4248
q11 773 695 714 695
q12 1044 836 860 836
q13 7211 3737 3755 3737
q14 451 435 435 435
q15 525 467 455 455
q16 738 684 691 684
q17 3841 3867 3893 3867
q18 8894 8826 8823 8823
q19 1744 1753 1673 1673
q20 2403 2160 2155 2155
q21 8711 8555 8587 8555
q22 1026 981 994 981
Total cold run time: 85275 ms
Total hot run time: 79778 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37598:
URL: https://github.com/apache/doris/pull/37598#issuecomment-2219793089
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) pick "[Fix](schema change) Fix schema change fault when add complex type column (#31824)" [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #37598:
URL: https://github.com/apache/doris/pull/37598
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org