You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "amorynan (via GitHub)" <gi...@apache.org> on 2024/07/08 12:42:15 UTC
[PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
amorynan opened a new pull request, #37495:
URL: https://github.com/apache/doris/pull/37495
## Proposed changes
when we use func array_with_const with large size there has not limit so, we will make be core like this
<img width="1512" alt="截屏2024-07-08 20 41 43" src="https://github.com/apache/doris/assets/18551114/01986f1f-5b25-48cb-89bb-c66e89b5c1d8">
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on code in PR #37495:
URL: https://github.com/apache/doris/pull/37495#discussion_r1674046341
##########
be/src/vec/functions/array/function_array_with_constant.cpp:
##########
@@ -91,9 +91,9 @@ class FunctionArrayWithConstant : public IFunction {
array_sizes.reserve(input_rows_count);
for (size_t i = 0; i < input_rows_count; ++i) {
auto array_size = num->get_int(i);
- if (UNLIKELY(array_size < 0)) {
- return Status::RuntimeError("Array size can not be negative in function:" +
- get_name());
+ if (UNLIKELY(array_size < 0) || UNLIKELY(array_size > max_array_size_as_field)) {
+ return Status::RuntimeError("Array size should in range(0, {}) in function: {}",
Review Comment:
add array_size actual value in error message for debug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on PR #37495:
URL: https://github.com/apache/doris/pull/37495#issuecomment-2213974384
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37495:
URL: https://github.com/apache/doris/pull/37495#issuecomment-2214089264
<details>
<summary>TPC-DS: <b>Total hot run time: 173284 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43d473c11b2041ce31a8f90baa30ab4c51f9a006, data reload: false
query1 920 377 366 366
query2 6457 2404 2446 2404
query3 6634 210 215 210
query4 28549 17370 17361 17361
query5 3636 506 486 486
query6 264 176 168 168
query7 4584 302 281 281
query8 303 282 286 282
query9 8388 2394 2368 2368
query10 554 290 271 271
query11 11940 10031 10070 10031
query12 120 82 86 82
query13 1650 373 369 369
query14 10114 7508 6607 6607
query15 242 190 187 187
query16 7746 293 299 293
query17 1566 529 523 523
query18 1945 275 269 269
query19 201 152 151 151
query20 95 106 80 80
query21 211 125 130 125
query22 4273 4109 4095 4095
query23 34099 33807 33737 33737
query24 11136 2866 2941 2866
query25 610 408 425 408
query26 717 150 143 143
query27 2252 272 275 272
query28 6291 2136 2121 2121
query29 886 633 618 618
query30 248 150 150 150
query31 1007 820 763 763
query32 96 52 55 52
query33 722 292 294 292
query34 997 480 507 480
query35 701 576 583 576
query36 1111 960 951 951
query37 156 87 84 84
query38 2951 2828 2794 2794
query39 889 842 849 842
query40 212 120 127 120
query41 60 53 54 53
query42 108 102 106 102
query43 625 513 535 513
query44 1166 711 713 711
query45 189 154 162 154
query46 1064 742 716 716
query47 1868 1812 1772 1772
query48 359 290 290 290
query49 846 405 405 405
query50 791 386 382 382
query51 6892 6746 6690 6690
query52 108 95 90 90
query53 362 287 283 283
query54 890 441 436 436
query55 74 102 73 73
query56 280 282 263 263
query57 1138 1052 1040 1040
query58 247 251 247 247
query59 3301 3232 3419 3232
query60 315 269 277 269
query61 98 94 97 94
query62 797 624 626 624
query63 328 290 280 280
query64 9097 2224 1709 1709
query65 3162 3104 3116 3104
query66 760 334 349 334
query67 15367 15035 14940 14940
query68 4776 531 533 531
query69 642 481 367 367
query70 1161 1120 1125 1120
query71 423 294 279 279
query72 7836 5315 5646 5315
query73 780 323 322 322
query74 5969 5503 5447 5447
query75 3518 2694 2652 2652
query76 3040 1012 959 959
query77 672 307 309 307
query78 9742 8995 8877 8877
query79 7573 521 504 504
query80 2749 481 473 473
query81 600 222 223 222
query82 1192 131 132 131
query83 304 166 159 159
query84 274 85 85 85
query85 1699 321 302 302
query86 470 304 318 304
query87 3285 3109 3066 3066
query88 5221 2395 2354 2354
query89 493 372 392 372
query90 1811 183 188 183
query91 131 102 100 100
query92 62 48 48 48
query93 5794 498 495 495
query94 1174 207 206 206
query95 398 315 308 308
query96 608 277 262 262
query97 3234 3030 3029 3029
query98 228 203 190 190
query99 1680 1231 1260 1231
Total cold run time: 293304 ms
Total hot run time: 173284 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37495:
URL: https://github.com/apache/doris/pull/37495#issuecomment-2213973969
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #37495:
URL: https://github.com/apache/doris/pull/37495#discussion_r1668749657
##########
regression-test/suites/datatype_p0/nested_types/query/array_functions/test_array_functions_array_with_const.groovy:
##########
@@ -16,6 +16,7 @@
// under the License.
suite("test_array_functions_array_with_const", "p0") {
+ sql "set enable_nereids_planner=false;"
Review Comment:
why set to false? Is Nererids planner ok for this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37495:
URL: https://github.com/apache/doris/pull/37495#issuecomment-2222262357
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #37495:
URL: https://github.com/apache/doris/pull/37495#discussion_r1674967255
##########
be/src/vec/functions/array/function_array_with_constant.cpp:
##########
@@ -91,9 +91,9 @@ class FunctionArrayWithConstant : public IFunction {
array_sizes.reserve(input_rows_count);
for (size_t i = 0; i < input_rows_count; ++i) {
auto array_size = num->get_int(i);
- if (UNLIKELY(array_size < 0)) {
- return Status::RuntimeError("Array size can not be negative in function:" +
- get_name());
+ if (UNLIKELY(array_size < 0) || UNLIKELY(array_size > max_array_size_as_field)) {
+ return Status::RuntimeError("Array size should in range(0, {}) in function: {}",
Review Comment:
array_size has already in error message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon merged PR #37495:
URL: https://github.com/apache/doris/pull/37495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37495:
URL: https://github.com/apache/doris/pull/37495#issuecomment-2214062094
<details>
<summary>TPC-H: <b>Total hot run time: 39926 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43d473c11b2041ce31a8f90baa30ab4c51f9a006, data reload: false
------ Round 1 ----------------------------------
q1 17627 4301 4264 4264
q2 2015 196 187 187
q3 10455 1195 1194 1194
q4 10183 799 815 799
q5 7466 2655 2621 2621
q6 223 142 140 140
q7 958 597 616 597
q8 9220 2086 2064 2064
q9 9044 6510 6457 6457
q10 9001 3677 3737 3677
q11 466 245 237 237
q12 475 234 237 234
q13 18753 2951 3006 2951
q14 263 224 219 219
q15 527 491 497 491
q16 515 390 376 376
q17 969 720 734 720
q18 8000 7430 7404 7404
q19 8438 1473 1480 1473
q20 692 317 325 317
q21 4903 3172 3286 3172
q22 383 337 332 332
Total cold run time: 120576 ms
Total hot run time: 39926 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4426 4259 4283 4259
q2 378 257 262 257
q3 3124 2972 2941 2941
q4 1975 1769 1744 1744
q5 5534 5563 5384 5384
q6 230 131 133 131
q7 2295 1882 1870 1870
q8 3322 3437 3397 3397
q9 8733 8864 8741 8741
q10 4132 3671 3845 3671
q11 583 494 502 494
q12 842 650 635 635
q13 17286 3100 3204 3100
q14 305 274 268 268
q15 526 493 500 493
q16 487 446 431 431
q17 1789 1543 1514 1514
q18 8137 8195 7864 7864
q19 1795 1608 1507 1507
q20 2192 1876 1827 1827
q21 5187 4863 4827 4827
q22 610 553 548 548
Total cold run time: 73888 ms
Total hot run time: 55903 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37495:
URL: https://github.com/apache/doris/pull/37495#issuecomment-2214102301
<details>
<summary>ClickBench: <b>Total hot run time: 30.58 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 43d473c11b2041ce31a8f90baa30ab4c51f9a006, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.03 0.04
query3 0.23 0.05 0.05
query4 1.68 0.08 0.08
query5 0.51 0.50 0.50
query6 1.13 0.73 0.72
query7 0.02 0.02 0.01
query8 0.05 0.04 0.05
query9 0.55 0.49 0.48
query10 0.55 0.55 0.54
query11 0.15 0.12 0.11
query12 0.14 0.12 0.12
query13 0.60 0.59 0.59
query14 0.76 0.78 0.78
query15 0.84 0.80 0.81
query16 0.38 0.37 0.36
query17 1.04 1.05 1.06
query18 0.22 0.21 0.21
query19 1.87 1.77 1.81
query20 0.01 0.01 0.01
query21 15.42 0.79 0.66
query22 4.28 6.83 1.81
query23 18.31 1.37 1.29
query24 2.10 0.23 0.21
query25 0.16 0.10 0.09
query26 0.29 0.21 0.21
query27 0.45 0.23 0.23
query28 13.33 1.02 1.00
query29 12.64 3.28 3.28
query30 0.25 0.05 0.07
query31 2.87 0.38 0.38
query32 3.27 0.47 0.47
query33 2.89 2.93 2.89
query34 17.02 4.39 4.33
query35 4.40 4.38 4.43
query36 0.66 0.46 0.49
query37 0.20 0.14 0.16
query38 0.15 0.15 0.14
query39 0.04 0.04 0.04
query40 0.15 0.12 0.13
query41 0.10 0.04 0.05
query42 0.05 0.05 0.04
query43 0.04 0.04 0.04
Total cold run time: 109.92 s
Total hot run time: 30.58 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37495:
URL: https://github.com/apache/doris/pull/37495#issuecomment-2222262425
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37495:
URL: https://github.com/apache/doris/pull/37495#issuecomment-2213986052
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "amorynan (via GitHub)" <gi...@apache.org>.
amorynan commented on code in PR #37495:
URL: https://github.com/apache/doris/pull/37495#discussion_r1669527557
##########
regression-test/suites/datatype_p0/nested_types/query/array_functions/test_array_functions_array_with_const.groovy:
##########
@@ -16,6 +16,7 @@
// under the License.
suite("test_array_functions_array_with_const", "p0") {
+ sql "set enable_nereids_planner=false;"
Review Comment:
below in this groovy we have nereids query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](func)fix array_with_const with larger than max_array_size [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on code in PR #37495:
URL: https://github.com/apache/doris/pull/37495#discussion_r1673829189
##########
regression-test/suites/datatype_p0/nested_types/query/array_functions/test_array_functions_array_with_const.groovy:
##########
@@ -31,7 +32,16 @@ suite("test_array_functions_array_with_const", "p0") {
SELECT array_with_constant(-231.37104, -138);
"""
} catch (Exception ex) {
- assertTrue(ex.getMessage().contains("Array size can not be negative in function:array_with_constant"))
+ assertTrue(ex.getMessage().contains("Array size should in range(0, 1000000) in function"))
+ }
+
+ // -- {server for large array}
+ try {
Review Comment:
don't use `try-catch` to do sql failure test. it doesn't work at all. if a sql didn't fail, the test will success. it's not as expected. please use `test{ sql exception }` instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org