You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yujun777 (via GitHub)" <gi...@apache.org> on 2024/02/08 08:29:43 UTC
[PR] [fix](dynamic partition) fix create too many dynamic partitions [doris]
yujun777 opened a new pull request, #30994:
URL: https://github.com/apache/doris/pull/30994
For dynamic partitions, the expect created partition num = dynamic_partition.end - dynamic_partition.start.
By default, fe limit 500(Config.max_dynamic_partition_num) dynamic partitions.
But if dynamic_partition.start is near int.min_value， the expect create partition num will be overflow and become a negative number. Then check created partition num < Config.max_dynamic_partition_num.
For example: if start = -2147483647 (int.min_value + 1)， end = 1, then end - start = -2147483648 < 0
So change the expect create partition num to long to avoid overflow.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](dynamic partition) fix create too many dynamic partitions [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30994:
URL: https://github.com/apache/doris/pull/30994#issuecomment-1933680933
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit c930391a084104cc289f820915012586e7bcb451 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.4 seconds inserted 10000000 Rows, about 694K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](dynamic partition) fix create too many dynamic partitions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #30994:
URL: https://github.com/apache/doris/pull/30994#issuecomment-1933616808
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](dynamic partition) fix create too many dynamic partitions [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30994:
URL: https://github.com/apache/doris/pull/30994#issuecomment-1933581025
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](dynamic partition) fix create too many dynamic partitions [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #30994:
URL: https://github.com/apache/doris/pull/30994#issuecomment-1933581198
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](dynamic partition) fix create too many dynamic partitions [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30994:
URL: https://github.com/apache/doris/pull/30994#issuecomment-1933667412
<details>
<summary>TPC-DS: <b>Total hot run time: 188659 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c930391a084104cc289f820915012586e7bcb451, data reload: false
query1 931 345 341 341
query2 6538 1818 1797 1797
query3 6698 206 198 198
query4 24640 22530 22402 22402
query5 4256 376 382 376
query6 270 163 164 163
query7 4612 289 291 289
query8 252 196 196 196
query9 8438 2809 2803 2803
query10 416 237 230 230
query11 16254 15629 15450 15450
query12 133 80 86 80
query13 1665 409 401 401
query14 9059 7631 7638 7631
query15 212 180 195 180
query16 7436 254 244 244
query17 1429 561 532 532
query18 1947 264 257 257
query19 191 144 144 144
query20 86 80 81 80
query21 190 121 114 114
query22 5075 4932 4828 4828
query23 32611 31551 31684 31551
query24 12788 3384 3419 3384
query25 647 375 355 355
query26 1824 150 155 150
query27 3060 322 306 306
query28 6596 1834 1823 1823
query29 1132 609 618 609
query30 275 134 154 134
query31 942 761 773 761
query32 95 57 54 54
query33 724 226 235 226
query34 1074 486 492 486
query35 936 844 824 824
query36 996 931 916 916
query37 131 61 61 61
query38 3477 3337 3312 3312
query39 1371 1319 1325 1319
query40 284 105 101 101
query41 36 34 35 34
query42 104 102 107 102
query43 485 490 454 454
query44 1055 684 698 684
query45 197 183 182 182
query46 1037 762 782 762
query47 1735 1614 1634 1614
query48 414 355 345 345
query49 1165 297 296 296
query50 794 371 385 371
query51 5329 5158 5151 5151
query52 106 88 88 88
query53 393 286 296 286
query54 299 221 228 221
query55 84 82 84 82
query56 210 200 201 200
query57 1089 972 993 972
query58 217 194 195 194
query59 2467 2281 2279 2279
query60 233 214 217 214
query61 82 82 88 82
query62 610 382 345 345
query63 323 282 281 281
query64 6383 3124 3075 3075
query65 3314 3267 3265 3265
query66 1348 322 325 322
query67 14570 14210 14165 14165
query68 5151 546 540 540
query69 518 356 351 351
query70 1286 1236 1208 1208
query71 442 256 255 255
query72 6281 2786 2601 2601
query73 713 311 308 308
query74 7139 6700 6692 6692
query75 3216 2580 2549 2549
query76 3257 1128 1192 1128
query77 362 234 231 231
query78 9403 8832 8830 8830
query79 994 518 510 510
query80 657 352 335 335
query81 440 208 207 207
query82 1301 84 85 84
query83 239 125 119 119
query84 227 77 78 77
query85 1158 339 324 324
query86 311 307 304 304
query87 3645 3403 3390 3390
query88 2721 2290 2285 2285
query89 435 355 352 352
query90 2094 165 168 165
query91 155 124 128 124
query92 52 49 48 48
query93 1021 487 491 487
query94 1117 187 183 183
query95 8459 8331 8494 8331
query96 574 270 267 267
query97 4467 4277 4295 4277
query98 220 205 192 192
query99 1101 738 731 731
Total cold run time: 282926 ms
Total hot run time: 188659 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](dynamic partition) fix create too many dynamic partitions [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30994:
URL: https://github.com/apache/doris/pull/30994#issuecomment-1933675989
<details>
<summary>ClickBench: <b>Total hot run time: 31 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c930391a084104cc289f820915012586e7bcb451, data reload: false
query1 0.02 0.02 0.02
query2 0.06 0.02 0.02
query3 0.23 0.07 0.08
query4 1.64 0.08 0.08
query5 0.49 0.49 0.48
query6 1.36 0.60 0.60
query7 0.01 0.01 0.02
query8 0.03 0.02 0.02
query9 0.53 0.45 0.46
query10 0.49 0.48 0.49
query11 0.12 0.10 0.10
query12 0.11 0.10 0.10
query13 0.58 0.59 0.59
query14 0.76 0.77 0.80
query15 0.82 0.81 0.79
query16 0.34 0.33 0.34
query17 0.88 0.90 0.93
query18 0.17 0.17 0.15
query19 1.79 1.70 1.68
query20 0.02 0.01 0.01
query21 15.41 0.58 0.59
query22 2.90 3.35 2.34
query23 17.31 1.09 1.12
query24 1.98 0.59 0.37
query25 0.65 0.06 0.05
query26 0.19 0.13 0.15
query27 0.06 0.06 0.05
query28 11.99 0.83 0.82
query29 12.60 3.42 3.35
query30 0.53 0.51 0.44
query31 2.77 0.36 0.36
query32 3.34 0.47 0.47
query33 3.13 3.13 3.11
query34 15.35 4.48 4.49
query35 4.51 4.49 4.49
query36 1.07 0.93 0.94
query37 0.07 0.05 0.06
query38 0.03 0.03 0.03
query39 0.02 0.02 0.02
query40 0.18 0.14 0.14
query41 0.07 0.01 0.02
query42 0.02 0.02 0.02
query43 0.02 0.02 0.02
Total cold run time: 104.65 s
Total hot run time: 31 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](dynamic partition) fix create too many dynamic partitions [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30994:
URL: https://github.com/apache/doris/pull/30994#issuecomment-1933644721
<details>
<summary>TPC-H: <b>Total hot run time: 41152 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c930391a084104cc289f820915012586e7bcb451, data reload: false
------ Round 1 ----------------------------------
q1 17618 4927 4838 4838
q2 2046 149 137 137
q3 10614 997 999 997
q4 4776 947 964 947
q5 7698 3184 3216 3184
q6 202 136 130 130
q7 1289 788 751 751
q8 9339 2061 2065 2061
q9 7608 6639 6681 6639
q10 8308 2657 2643 2643
q11 409 210 203 203
q12 749 325 324 324
q13 17994 3686 3676 3676
q14 288 258 253 253
q15 591 511 503 503
q16 501 402 409 402
q17 914 862 770 770
q18 7490 6592 6609 6592
q19 1560 1484 1487 1484
q20 595 357 340 340
q21 7007 3962 3933 3933
q22 865 349 345 345
Total cold run time: 108461 ms
Total hot run time: 41152 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4844 4820 4832 4820
q2 297 179 181 179
q3 3582 3555 3570 3555
q4 2504 2514 2505 2505
q5 5776 5752 5740 5740
q6 209 128 128 128
q7 2231 1655 1636 1636
q8 3003 3064 3091 3064
q9 8687 8677 8722 8677
q10 6756 4211 4201 4201
q11 506 377 359 359
q12 760 527 539 527
q13 4990 3374 3377 3374
q14 261 248 245 245
q15 593 493 488 488
q16 488 437 433 433
q17 1656 1593 1598 1593
q18 8344 7580 7681 7580
q19 1635 1640 1635 1635
q20 2111 1826 1825 1825
q21 6518 6093 6087 6087
q22 582 524 504 504
Total cold run time: 66333 ms
Total hot run time: 59155 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org