You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "liujiwen-up (via GitHub)" <gi...@apache.org> on 2024/10/10 14:46:38 UTC
[PR] [feature](functions) impl scalar functions trim_in [doris]
liujiwen-up opened a new pull request, #41681:
URL: https://github.com/apache/doris/pull/41681
trim_in is different from trim
Find and remove any characters in a set of characters at both ends of a string (regardless of order)
mysql> SELECT TRIM('abcd', 'cde');
+---------------------+
| trim('abcd', 'cde') |
+---------------------+
| abcd |
+---------------------+
1 row in set (0.02 sec)
mysql> SELECT TRIM_IN('abcd', 'cde');
+------------------------+
| trim_in('abcd', 'cde') |
+------------------------+
| ab |
+------------------------+
1 row in set (0.02 sec)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1802484040
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -640,6 +788,38 @@ class FunctionTrim : public IFunction {
}
};
+template <typename impl>
Review Comment:
done
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -583,14 +722,23 @@ struct Trim2Impl {
const auto* remove_str_raw = col_right->get_chars().data();
const ColumnString::Offset remove_str_size = col_right->get_offsets()[0];
const StringRef remove_str(remove_str_raw, remove_str_size);
+
if (remove_str.size == 1) {
RETURN_IF_ERROR((TrimUtil<is_ltrim, is_rtrim, true>::vector(
col->get_chars(), col->get_offsets(), remove_str, col_res->get_chars(),
col_res->get_offsets())));
} else {
- RETURN_IF_ERROR((TrimUtil<is_ltrim, is_rtrim, false>::vector(
- col->get_chars(), col->get_offsets(), remove_str, col_res->get_chars(),
- col_res->get_offsets())));
+ if constexpr (std::is_same<Name, NameTrimIn>::value ||
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2413320378
TeamCity be ut coverage result:
Function Coverage: 37.46% (9711/25926)
Line Coverage: 28.73% (80635/280623)
Region Coverage: 28.18% (41719/148033)
Branch Coverage: 24.76% (21209/85670)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f6894976d3ee54eccae720981cbfa97f982a5084_f6894976d3ee54eccae720981cbfa97f982a5084/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2416594946
<details>
<summary>TPC-DS: <b>Total hot run time: 192384 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c741d6d007719db8af2bfdb3cbb9af8f2a05f531, data reload: false
query1 899 403 416 403
query2 6275 2046 1978 1978
query3 8695 194 204 194
query4 34270 23735 23671 23671
query5 3475 488 479 479
query6 264 162 174 162
query7 4200 289 286 286
query8 293 226 251 226
query9 9470 2687 2707 2687
query10 468 279 269 269
query11 18048 15173 15309 15173
query12 157 101 100 100
query13 1560 428 417 417
query14 9570 7267 7793 7267
query15 252 168 168 168
query16 7445 436 476 436
query17 1654 614 581 581
query18 1441 310 307 307
query19 383 158 173 158
query20 127 114 118 114
query21 208 105 105 105
query22 5078 4859 4831 4831
query23 34985 34292 34337 34292
query24 10993 2734 2726 2726
query25 585 424 413 413
query26 1153 160 163 160
query27 2342 280 284 280
query28 7485 2407 2410 2407
query29 745 436 422 422
query30 266 158 157 157
query31 1031 793 811 793
query32 101 54 54 54
query33 770 289 291 289
query34 934 503 515 503
query35 903 739 724 724
query36 1103 949 955 949
query37 156 101 88 88
query38 3996 3922 3927 3922
query39 1455 1434 1418 1418
query40 252 101 101 101
query41 48 50 45 45
query42 123 99 96 96
query43 516 498 476 476
query44 1254 791 787 787
query45 197 167 173 167
query46 1127 710 703 703
query47 2004 1863 1883 1863
query48 429 321 321 321
query49 955 440 426 426
query50 807 383 399 383
query51 7173 7033 6915 6915
query52 107 94 90 90
query53 259 181 178 178
query54 1265 453 427 427
query55 83 79 76 76
query56 290 287 257 257
query57 1315 1131 1145 1131
query58 229 226 235 226
query59 3083 2954 2851 2851
query60 289 273 270 270
query61 106 98 114 98
query62 882 661 690 661
query63 220 186 180 180
query64 5026 635 607 607
query65 3259 3222 3209 3209
query66 1288 301 309 301
query67 16162 15772 15775 15772
query68 4454 557 564 557
query69 502 290 318 290
query70 1204 1121 1139 1121
query71 375 272 272 272
query72 7176 3970 3939 3939
query73 787 357 372 357
query74 10543 9023 8998 8998
query75 3401 2701 2621 2621
query76 3143 921 915 915
query77 417 301 287 287
query78 10421 9657 9693 9657
query79 2674 601 613 601
query80 2742 433 452 433
query81 578 255 240 240
query82 707 140 136 136
query83 306 139 142 139
query84 280 70 71 70
query85 1422 312 287 287
query86 397 298 299 298
query87 4632 4387 4232 4232
query88 4307 2285 2161 2161
query89 400 278 294 278
query90 2021 191 189 189
query91 137 103 102 102
query92 60 46 48 46
query93 1766 537 534 534
query94 1117 302 289 289
query95 357 245 254 245
query96 608 287 287 287
query97 3270 3181 3091 3091
query98 222 200 198 198
query99 1581 1288 1293 1288
Total cold run time: 303990 ms
Total hot run time: 192384 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2413356024
<details>
<summary>TPC-DS: <b>Total hot run time: 191690 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f6894976d3ee54eccae720981cbfa97f982a5084, data reload: false
query1 942 411 444 411
query2 6255 2109 2048 2048
query3 8684 196 200 196
query4 34244 23752 24044 23752
query5 3556 487 463 463
query6 292 175 167 167
query7 4189 296 303 296
query8 303 238 236 236
query9 9150 2731 2722 2722
query10 446 276 273 273
query11 17945 15368 15349 15349
query12 152 102 107 102
query13 1571 444 421 421
query14 9364 7057 7443 7057
query15 250 177 176 176
query16 7995 450 488 450
query17 1662 614 590 590
query18 2077 298 324 298
query19 379 156 157 156
query20 122 116 115 115
query21 208 107 109 107
query22 4725 4695 4449 4449
query23 35289 34306 33973 33973
query24 10882 2674 2739 2674
query25 622 412 393 393
query26 1196 160 171 160
query27 2263 284 283 283
query28 7568 2406 2411 2406
query29 842 433 424 424
query30 260 157 153 153
query31 1060 807 814 807
query32 101 63 64 63
query33 767 298 301 298
query34 910 494 540 494
query35 904 751 739 739
query36 1109 930 965 930
query37 155 85 89 85
query38 4039 3953 3829 3829
query39 1501 1432 1427 1427
query40 208 101 99 99
query41 48 46 47 46
query42 122 106 101 101
query43 528 492 485 485
query44 1238 819 804 804
query45 201 167 167 167
query46 1143 702 716 702
query47 1964 1839 1845 1839
query48 412 322 331 322
query49 950 440 438 438
query50 820 391 396 391
query51 7060 6989 6984 6984
query52 99 86 89 86
query53 254 176 178 176
query54 1147 425 423 423
query55 78 77 77 77
query56 268 267 265 265
query57 1284 1155 1160 1155
query58 233 237 234 234
query59 3194 2981 2729 2729
query60 286 263 264 263
query61 105 104 102 102
query62 864 662 665 662
query63 214 182 180 180
query64 4020 653 621 621
query65 3303 3210 3216 3210
query66 953 296 343 296
query67 16002 15649 15637 15637
query68 4551 556 563 556
query69 536 292 289 289
query70 1186 1144 1134 1134
query71 369 269 280 269
query72 7510 4016 3988 3988
query73 772 356 354 354
query74 10226 8984 9068 8984
query75 3454 2701 2698 2698
query76 3155 932 842 842
query77 606 297 297 297
query78 10591 9662 9597 9597
query79 1879 587 590 587
query80 1159 445 455 445
query81 590 242 237 237
query82 682 137 139 137
query83 296 136 146 136
query84 278 73 68 68
query85 1413 298 287 287
query86 431 303 277 277
query87 4622 4324 4343 4324
query88 3332 2198 2165 2165
query89 417 287 287 287
query90 1997 183 183 183
query91 153 106 100 100
query92 72 46 53 46
query93 2229 531 533 531
query94 1074 284 281 281
query95 358 246 241 241
query96 621 282 290 282
query97 3327 3125 3175 3125
query98 217 209 194 194
query99 1563 1283 1314 1283
Total cold run time: 300643 ms
Total hot run time: 191690 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1796789066
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -535,6 +545,42 @@ struct TrimUtil {
return Status::OK();
}
};
+template <bool is_ltrim_in, bool is_rtrim_in, bool trim_single>
+struct TrimInUtil {
+ static Status vector(const ColumnString::Chars& str_data,
Review Comment:
In order to maintain the same processing method as the TrimUtil method, so no changes are made here.
`struct TrimUtil {
static Status vector(const ColumnString::Chars& str_data,
const ColumnString::Offsets& str_offsets, const StringRef& remove_str,
ColumnString::Chars& res_data, ColumnString::Offsets& res_offsets)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2415628091
<details>
<summary>ClickBench: <b>Total hot run time: 32.76 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 90febeb9c3d1eb0ab8100cf4ac008b9e55dc8412, data reload: false
query1 0.04 0.03 0.02
query2 0.07 0.03 0.03
query3 0.22 0.07 0.07
query4 1.65 0.09 0.10
query5 0.52 0.52 0.54
query6 1.13 0.73 0.73
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.56 0.49 0.48
query10 0.55 0.56 0.53
query11 0.14 0.11 0.10
query12 0.14 0.11 0.11
query13 0.61 0.60 0.61
query14 2.73 2.80 2.72
query15 0.91 0.83 0.83
query16 0.39 0.35 0.36
query17 1.07 1.06 0.97
query18 0.23 0.23 0.21
query19 1.98 1.86 2.06
query20 0.01 0.02 0.01
query21 15.62 0.61 0.56
query22 2.40 2.26 1.91
query23 16.92 0.94 0.85
query24 3.45 1.44 1.00
query25 0.13 0.25 0.11
query26 0.47 0.14 0.12
query27 0.05 0.05 0.04
query28 10.40 1.08 1.06
query29 12.56 3.19 3.17
query30 0.26 0.06 0.07
query31 2.85 0.38 0.36
query32 3.28 0.46 0.45
query33 2.97 3.03 3.06
query34 17.07 4.49 4.45
query35 4.55 4.65 4.53
query36 0.67 0.49 0.48
query37 0.08 0.06 0.06
query38 0.05 0.04 0.03
query39 0.03 0.02 0.03
query40 0.17 0.13 0.13
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 107.13 s
Total hot run time: 32.76 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1801173448
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -535,6 +545,135 @@ struct TrimUtil {
return Status::OK();
}
};
+template <bool is_ltrim_in, bool is_rtrim_in, bool trim_single>
+struct TrimInUtil {
+ static Status vector(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets, const StringRef& remove_str,
+ ColumnString::Chars& res_data, ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ res_offsets.resize(offset_size);
+ res_data.reserve(str_data.size());
+ bool all_ascii = simd::VStringFunctions::is_ascii(remove_str) &&
+ simd::VStringFunctions::is_ascii(StringRef(
+ reinterpret_cast<const char*>(str_data.data()), str_data.size()));
+
+ if (all_ascii) {
+ return impl_vectors_ascii(str_data, str_offsets, remove_str, res_data, res_offsets);
+ } else {
+ return impl_vectors_utf8(str_data, str_offsets, remove_str, res_data, res_offsets);
+ }
+ }
+
+private:
+ static Status impl_vectors_ascii(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets,
+ const StringRef& remove_str, ColumnString::Chars& res_data,
+ ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ std::bitset<128> char_lookup;
+ const char* remove_begin = remove_str.data;
+ const char* remove_end = remove_str.data + remove_str.size;
+
+ while (remove_begin < remove_end) {
+ char_lookup.set(static_cast<unsigned char>(*remove_begin));
+ remove_begin += 1;
+ }
+
+ for (size_t i = 0; i < offset_size; ++i) {
+ const char* str_begin = reinterpret_cast<const char*>(
+ str_data.data() + (i == 0 ? 0 : str_offsets[i - 1]));
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2413326050
<details>
<summary>TPC-H: <b>Total hot run time: 41467 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f6894976d3ee54eccae720981cbfa97f982a5084, data reload: false
------ Round 1 ----------------------------------
q1 17583 7476 7350 7350
q2 2029 297 279 279
q3 12157 1076 1214 1076
q4 10575 827 756 756
q5 7772 3144 3071 3071
q6 252 153 148 148
q7 1028 624 594 594
q8 9356 1998 1930 1930
q9 6599 6486 6427 6427
q10 7070 2426 2481 2426
q11 439 250 253 250
q12 409 224 222 222
q13 17796 3037 3025 3025
q14 253 214 220 214
q15 571 535 532 532
q16 636 592 589 589
q17 975 564 503 503
q18 7158 6815 6696 6696
q19 1349 1078 1057 1057
q20 488 188 185 185
q21 4043 3136 3244 3136
q22 1101 1001 1021 1001
Total cold run time: 109639 ms
Total hot run time: 41467 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7380 7271 7274 7271
q2 336 228 238 228
q3 3064 2986 2966 2966
q4 2140 1859 1823 1823
q5 5743 5739 5841 5739
q6 238 141 146 141
q7 2270 1822 1813 1813
q8 3405 3526 3474 3474
q9 8973 8929 8916 8916
q10 3604 3584 3572 3572
q11 596 504 478 478
q12 847 684 630 630
q13 8448 3207 3185 3185
q14 315 272 293 272
q15 579 528 527 527
q16 696 650 646 646
q17 1841 1634 1623 1623
q18 8351 7704 7497 7497
q19 1753 1482 1519 1482
q20 2113 1870 1910 1870
q21 5570 5398 5307 5307
q22 1142 1055 1044 1044
Total cold run time: 69404 ms
Total hot run time: 60504 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2419112713
> Please add doc in https://github.com/apache/doris-website @liujiwen-up
https://github.com/apache/doris-website/pull/1207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2413820396
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1802314314
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -583,14 +722,23 @@ struct Trim2Impl {
const auto* remove_str_raw = col_right->get_chars().data();
const ColumnString::Offset remove_str_size = col_right->get_offsets()[0];
const StringRef remove_str(remove_str_raw, remove_str_size);
+
if (remove_str.size == 1) {
RETURN_IF_ERROR((TrimUtil<is_ltrim, is_rtrim, true>::vector(
col->get_chars(), col->get_offsets(), remove_str, col_res->get_chars(),
col_res->get_offsets())));
} else {
- RETURN_IF_ERROR((TrimUtil<is_ltrim, is_rtrim, false>::vector(
- col->get_chars(), col->get_offsets(), remove_str, col_res->get_chars(),
- col_res->get_offsets())));
+ if constexpr (std::is_same<Name, NameTrimIn>::value ||
Review Comment:
`TrimInUtil` and `TrimUtil` better be a template param
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2416607244
<details>
<summary>ClickBench: <b>Total hot run time: 32.53 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c741d6d007719db8af2bfdb3cbb9af8f2a05f531, data reload: false
query1 0.04 0.04 0.03
query2 0.06 0.03 0.02
query3 0.24 0.07 0.07
query4 1.65 0.11 0.11
query5 0.54 0.51 0.53
query6 1.14 0.73 0.72
query7 0.02 0.01 0.02
query8 0.04 0.04 0.03
query9 0.59 0.51 0.49
query10 0.56 0.54 0.55
query11 0.13 0.10 0.10
query12 0.14 0.11 0.10
query13 0.60 0.60 0.59
query14 2.71 2.79 2.80
query15 0.89 0.83 0.85
query16 0.37 0.38 0.38
query17 0.97 1.05 1.08
query18 0.20 0.19 0.20
query19 1.93 1.84 1.98
query20 0.01 0.01 0.01
query21 15.35 0.56 0.57
query22 2.66 1.81 2.04
query23 16.88 0.99 0.92
query24 3.39 0.75 1.61
query25 0.20 0.34 0.09
query26 0.44 0.13 0.13
query27 0.04 0.05 0.04
query28 10.53 1.10 1.05
query29 12.55 3.25 3.23
query30 0.24 0.07 0.06
query31 2.86 0.40 0.37
query32 3.28 0.46 0.46
query33 2.99 2.99 3.03
query34 16.96 4.51 4.42
query35 4.49 4.49 4.48
query36 0.66 0.51 0.49
query37 0.09 0.05 0.05
query38 0.05 0.04 0.03
query39 0.03 0.02 0.02
query40 0.17 0.12 0.11
query41 0.08 0.02 0.02
query42 0.03 0.02 0.01
query43 0.03 0.04 0.03
Total cold run time: 106.83 s
Total hot run time: 32.53 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2418747356
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2406720142
TeamCity be ut coverage result:
Function Coverage: 37.40% (9661/25833)
Line Coverage: 28.68% (80179/279544)
Region Coverage: 28.10% (41440/147483)
Branch Coverage: 24.71% (21107/85432)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4452a4b5f4f3e6d4fe4418cccca7ba45f97012c2_4452a4b5f4f3e6d4fe4418cccca7ba45f97012c2/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2408306886
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1802296304
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -640,6 +788,38 @@ class FunctionTrim : public IFunction {
}
};
+template <typename impl>
Review Comment:
Impl and the `FunctionTrimIn` code is same as `FunctionTrim`. should reuse not imple another
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2416480616
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2413246692
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2413808986
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1801174551
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -535,6 +545,135 @@ struct TrimUtil {
return Status::OK();
}
};
+template <bool is_ltrim_in, bool is_rtrim_in, bool trim_single>
+struct TrimInUtil {
+ static Status vector(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets, const StringRef& remove_str,
+ ColumnString::Chars& res_data, ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ res_offsets.resize(offset_size);
+ res_data.reserve(str_data.size());
+ bool all_ascii = simd::VStringFunctions::is_ascii(remove_str) &&
+ simd::VStringFunctions::is_ascii(StringRef(
+ reinterpret_cast<const char*>(str_data.data()), str_data.size()));
+
+ if (all_ascii) {
+ return impl_vectors_ascii(str_data, str_offsets, remove_str, res_data, res_offsets);
+ } else {
+ return impl_vectors_utf8(str_data, str_offsets, remove_str, res_data, res_offsets);
+ }
+ }
+
+private:
+ static Status impl_vectors_ascii(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets,
+ const StringRef& remove_str, ColumnString::Chars& res_data,
+ ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ std::bitset<128> char_lookup;
+ const char* remove_begin = remove_str.data;
+ const char* remove_end = remove_str.data + remove_str.size;
+
+ while (remove_begin < remove_end) {
+ char_lookup.set(static_cast<unsigned char>(*remove_begin));
+ remove_begin += 1;
+ }
+
+ for (size_t i = 0; i < offset_size; ++i) {
+ const char* str_begin = reinterpret_cast<const char*>(
+ str_data.data() + (i == 0 ? 0 : str_offsets[i - 1]));
+ const char* str_end = reinterpret_cast<const char*>(str_data.data() + str_offsets[i]);
+ const char* left_trim_pos = str_begin;
+ const char* right_trim_pos = str_end;
+
+ if constexpr (is_ltrim_in) {
+ while (left_trim_pos < str_end) {
+ if (!char_lookup.test(static_cast<unsigned char>(*left_trim_pos))) {
+ break;
+ }
+ ++left_trim_pos;
+ }
+ }
+
+ if constexpr (is_rtrim_in) {
+ while (right_trim_pos > left_trim_pos) {
+ --right_trim_pos;
+ if (!char_lookup.test(static_cast<unsigned char>(*right_trim_pos))) {
+ ++right_trim_pos;
+ break;
+ }
+ }
+ }
+
+ res_data.insert_assume_reserved(left_trim_pos, right_trim_pos);
+ res_offsets[i] = res_data.size();
+ }
+
+ return Status::OK();
+ }
+
+ static Status impl_vectors_utf8(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets,
+ const StringRef& remove_str, ColumnString::Chars& res_data,
+ ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ res_offsets.resize(offset_size);
+ res_data.reserve(str_data.size());
+
+ std::unordered_set<std::string> char_lookup;
Review Comment:
Change to string_view
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2416560138
TeamCity be ut coverage result:
Function Coverage: 37.45% (9709/25922)
Line Coverage: 28.72% (80637/280737)
Region Coverage: 28.15% (41707/148140)
Branch Coverage: 24.74% (21211/85732)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c741d6d007719db8af2bfdb3cbb9af8f2a05f531_c741d6d007719db8af2bfdb3cbb9af8f2a05f531/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2418817502
TeamCity be ut coverage result:
Function Coverage: 37.46% (9710/25922)
Line Coverage: 28.72% (80634/280729)
Region Coverage: 28.17% (41725/148139)
Branch Coverage: 24.74% (21210/85730)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c741d6d007719db8af2bfdb3cbb9af8f2a05f531_c741d6d007719db8af2bfdb3cbb9af8f2a05f531/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 merged PR #41681:
URL: https://github.com/apache/doris/pull/41681
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2421167564
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2406678071
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1802296304
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -640,6 +788,38 @@ class FunctionTrim : public IFunction {
}
};
+template <typename impl>
Review Comment:
Impl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2416564008
<details>
<summary>TPC-H: <b>Total hot run time: 41398 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c741d6d007719db8af2bfdb3cbb9af8f2a05f531, data reload: false
------ Round 1 ----------------------------------
q1 17620 7391 7276 7276
q2 2011 295 282 282
q3 12117 1025 1149 1025
q4 10583 902 905 902
q5 7756 3106 3081 3081
q6 241 151 148 148
q7 1032 627 630 627
q8 9373 1925 2021 1925
q9 6588 6443 6420 6420
q10 7019 2407 2462 2407
q11 448 251 239 239
q12 412 217 223 217
q13 17760 2963 2963 2963
q14 246 207 221 207
q15 583 533 526 526
q16 660 578 574 574
q17 962 542 592 542
q18 7248 6796 6821 6796
q19 1336 1036 896 896
q20 478 185 183 183
q21 4017 3332 3150 3150
q22 1119 1012 1030 1012
Total cold run time: 109609 ms
Total hot run time: 41398 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7266 7239 7244 7239
q2 328 226 228 226
q3 3069 3015 2980 2980
q4 2022 1833 1779 1779
q5 5725 5780 5775 5775
q6 238 144 142 142
q7 2281 1826 1769 1769
q8 3395 3528 3467 3467
q9 8917 8909 8823 8823
q10 3627 3561 3562 3561
q11 593 487 493 487
q12 854 655 652 652
q13 10715 3207 3213 3207
q14 313 268 270 268
q15 578 535 518 518
q16 685 637 635 635
q17 1831 1636 1613 1613
q18 8333 7831 7639 7639
q19 1703 1400 1351 1351
q20 2097 1904 1859 1859
q21 5534 5522 5381 5381
q22 1111 1087 1075 1075
Total cold run time: 71215 ms
Total hot run time: 60446 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2411545554
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1797631130
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -535,6 +545,42 @@ struct TrimUtil {
return Status::OK();
}
};
+template <bool is_ltrim_in, bool is_rtrim_in, bool trim_single>
+struct TrimInUtil {
+ static Status vector(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets, const StringRef& remove_str,
+ ColumnString::Chars& res_data, ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ res_offsets.resize(offset_size);
+ res_data.reserve(str_data.size());
+ std::bitset<256> char_lookup;
Review Comment:
Add comment to explain magic number 256
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -535,6 +545,42 @@ struct TrimUtil {
return Status::OK();
}
};
+template <bool is_ltrim_in, bool is_rtrim_in, bool trim_single>
+struct TrimInUtil {
+ static Status vector(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets, const StringRef& remove_str,
+ ColumnString::Chars& res_data, ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ res_offsets.resize(offset_size);
+ res_data.reserve(str_data.size());
+ std::bitset<256> char_lookup;
+ for (size_t i = 0; i < remove_str.size; ++i) {
+ char_lookup[static_cast<unsigned char>(remove_str.data[i])] = true;
Review Comment:
what if chinese
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2413368417
<details>
<summary>ClickBench: <b>Total hot run time: 33.23 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f6894976d3ee54eccae720981cbfa97f982a5084, data reload: false
query1 0.04 0.04 0.03
query2 0.07 0.03 0.04
query3 0.23 0.06 0.06
query4 1.64 0.10 0.10
query5 0.52 0.52 0.51
query6 1.16 0.72 0.73
query7 0.03 0.02 0.01
query8 0.04 0.03 0.03
query9 0.57 0.51 0.50
query10 0.55 0.55 0.55
query11 0.14 0.11 0.11
query12 0.14 0.11 0.11
query13 0.61 0.62 0.60
query14 2.70 2.77 2.71
query15 0.91 0.82 0.85
query16 0.38 0.38 0.38
query17 0.98 1.06 0.99
query18 0.20 0.19 0.20
query19 1.85 1.79 1.87
query20 0.02 0.01 0.01
query21 15.37 0.58 0.56
query22 2.27 2.01 1.56
query23 16.94 0.84 0.92
query24 3.06 1.88 1.68
query25 0.28 0.13 0.17
query26 0.52 0.13 0.14
query27 0.04 0.03 0.03
query28 9.41 1.11 1.07
query29 12.55 3.25 3.25
query30 0.24 0.06 0.06
query31 2.86 0.38 0.38
query32 3.26 0.46 0.45
query33 3.00 3.06 3.06
query34 17.08 4.45 4.49
query35 4.54 4.54 4.54
query36 0.65 0.51 0.50
query37 0.09 0.06 0.06
query38 0.05 0.03 0.03
query39 0.03 0.02 0.02
query40 0.17 0.12 0.12
query41 0.08 0.02 0.03
query42 0.03 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 105.34 s
Total hot run time: 33.23 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1800577717
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -535,6 +545,135 @@ struct TrimUtil {
return Status::OK();
}
};
+template <bool is_ltrim_in, bool is_rtrim_in, bool trim_single>
+struct TrimInUtil {
+ static Status vector(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets, const StringRef& remove_str,
+ ColumnString::Chars& res_data, ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ res_offsets.resize(offset_size);
+ res_data.reserve(str_data.size());
+ bool all_ascii = simd::VStringFunctions::is_ascii(remove_str) &&
+ simd::VStringFunctions::is_ascii(StringRef(
+ reinterpret_cast<const char*>(str_data.data()), str_data.size()));
+
+ if (all_ascii) {
+ return impl_vectors_ascii(str_data, str_offsets, remove_str, res_data, res_offsets);
+ } else {
+ return impl_vectors_utf8(str_data, str_offsets, remove_str, res_data, res_offsets);
+ }
+ }
+
+private:
+ static Status impl_vectors_ascii(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets,
+ const StringRef& remove_str, ColumnString::Chars& res_data,
+ ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ std::bitset<128> char_lookup;
+ const char* remove_begin = remove_str.data;
+ const char* remove_end = remove_str.data + remove_str.size;
+
+ while (remove_begin < remove_end) {
+ char_lookup.set(static_cast<unsigned char>(*remove_begin));
+ remove_begin += 1;
+ }
+
+ for (size_t i = 0; i < offset_size; ++i) {
+ const char* str_begin = reinterpret_cast<const char*>(
+ str_data.data() + (i == 0 ? 0 : str_offsets[i - 1]));
Review Comment:
`str_data.data() + str_offsets[i]` is better, since PODArray guarantees access to -1 is safe.
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -535,6 +545,135 @@ struct TrimUtil {
return Status::OK();
}
};
+template <bool is_ltrim_in, bool is_rtrim_in, bool trim_single>
+struct TrimInUtil {
+ static Status vector(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets, const StringRef& remove_str,
+ ColumnString::Chars& res_data, ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ res_offsets.resize(offset_size);
+ res_data.reserve(str_data.size());
+ bool all_ascii = simd::VStringFunctions::is_ascii(remove_str) &&
+ simd::VStringFunctions::is_ascii(StringRef(
+ reinterpret_cast<const char*>(str_data.data()), str_data.size()));
+
+ if (all_ascii) {
+ return impl_vectors_ascii(str_data, str_offsets, remove_str, res_data, res_offsets);
+ } else {
+ return impl_vectors_utf8(str_data, str_offsets, remove_str, res_data, res_offsets);
+ }
+ }
+
+private:
+ static Status impl_vectors_ascii(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets,
+ const StringRef& remove_str, ColumnString::Chars& res_data,
+ ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ std::bitset<128> char_lookup;
+ const char* remove_begin = remove_str.data;
+ const char* remove_end = remove_str.data + remove_str.size;
+
+ while (remove_begin < remove_end) {
+ char_lookup.set(static_cast<unsigned char>(*remove_begin));
+ remove_begin += 1;
+ }
+
+ for (size_t i = 0; i < offset_size; ++i) {
+ const char* str_begin = reinterpret_cast<const char*>(
+ str_data.data() + (i == 0 ? 0 : str_offsets[i - 1]));
+ const char* str_end = reinterpret_cast<const char*>(str_data.data() + str_offsets[i]);
+ const char* left_trim_pos = str_begin;
+ const char* right_trim_pos = str_end;
+
+ if constexpr (is_ltrim_in) {
+ while (left_trim_pos < str_end) {
+ if (!char_lookup.test(static_cast<unsigned char>(*left_trim_pos))) {
+ break;
+ }
+ ++left_trim_pos;
+ }
+ }
+
+ if constexpr (is_rtrim_in) {
+ while (right_trim_pos > left_trim_pos) {
+ --right_trim_pos;
+ if (!char_lookup.test(static_cast<unsigned char>(*right_trim_pos))) {
+ ++right_trim_pos;
+ break;
+ }
+ }
+ }
+
+ res_data.insert_assume_reserved(left_trim_pos, right_trim_pos);
+ res_offsets[i] = res_data.size();
+ }
+
+ return Status::OK();
+ }
+
+ static Status impl_vectors_utf8(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets,
+ const StringRef& remove_str, ColumnString::Chars& res_data,
+ ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ res_offsets.resize(offset_size);
+ res_data.reserve(str_data.size());
+
+ std::unordered_set<std::string> char_lookup;
Review Comment:
potential large momory comsuption
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2413244356
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1800408736
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -535,6 +545,42 @@ struct TrimUtil {
return Status::OK();
}
};
+template <bool is_ltrim_in, bool is_rtrim_in, bool trim_single>
+struct TrimInUtil {
+ static Status vector(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets, const StringRef& remove_str,
+ ColumnString::Chars& res_data, ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ res_offsets.resize(offset_size);
+ res_data.reserve(str_data.size());
+ std::bitset<256> char_lookup;
+ for (size_t i = 0; i < remove_str.size; ++i) {
+ char_lookup[static_cast<unsigned char>(remove_str.data[i])] = true;
Review Comment:
Added utf-8 processing logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2405320393
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2418845236
<details>
<summary>TPC-DS: <b>Total hot run time: 191866 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c741d6d007719db8af2bfdb3cbb9af8f2a05f531, data reload: false
query1 914 391 401 391
query2 6243 2113 2048 2048
query3 8685 205 206 205
query4 34155 23767 23711 23711
query5 3491 484 458 458
query6 262 169 163 163
query7 4194 297 308 297
query8 301 237 220 220
query9 9356 2700 2694 2694
query10 462 277 266 266
query11 18059 15296 15351 15296
query12 167 105 109 105
query13 1567 461 416 416
query14 9723 7073 7003 7003
query15 247 170 182 170
query16 7997 407 486 407
query17 1610 611 613 611
query18 2149 333 310 310
query19 308 156 162 156
query20 124 123 116 116
query21 206 112 111 111
query22 4628 4478 4548 4478
query23 35126 33875 34148 33875
query24 11058 2769 2749 2749
query25 621 421 412 412
query26 1059 162 167 162
query27 2383 293 283 283
query28 7591 2423 2417 2417
query29 808 450 426 426
query30 257 157 167 157
query31 1116 802 818 802
query32 101 57 59 57
query33 758 306 306 306
query34 941 504 530 504
query35 866 773 770 770
query36 1102 959 949 949
query37 159 91 88 88
query38 4230 3972 3900 3900
query39 1617 1425 1435 1425
query40 208 102 100 100
query41 49 51 48 48
query42 131 109 112 109
query43 542 503 479 479
query44 1236 810 811 810
query45 200 170 172 170
query46 1124 716 728 716
query47 1967 1817 1832 1817
query48 446 321 320 320
query49 964 435 417 417
query50 838 391 416 391
query51 7164 6995 6879 6879
query52 106 90 92 90
query53 259 182 184 182
query54 1257 453 449 449
query55 82 77 79 77
query56 285 277 277 277
query57 1293 1171 1164 1164
query58 267 232 274 232
query59 3156 2939 2941 2939
query60 299 262 272 262
query61 103 99 100 99
query62 846 677 687 677
query63 235 191 185 185
query64 4014 632 598 598
query65 3324 3232 3224 3224
query66 829 310 307 307
query67 16009 15721 15444 15444
query68 4467 580 554 554
query69 519 291 300 291
query70 1202 1127 1138 1127
query71 357 294 280 280
query72 7310 4122 3983 3983
query73 798 364 373 364
query74 10601 9040 9127 9040
query75 3436 2676 2669 2669
query76 2896 896 883 883
query77 649 307 296 296
query78 10596 9619 9682 9619
query79 1962 647 621 621
query80 1131 453 476 453
query81 576 246 241 241
query82 697 149 144 144
query83 296 137 136 136
query84 285 69 70 69
query85 1464 324 289 289
query86 395 285 283 283
query87 4586 4320 4403 4320
query88 2946 2241 2184 2184
query89 409 289 288 288
query90 1926 194 190 190
query91 140 106 103 103
query92 67 49 51 49
query93 1873 538 534 534
query94 989 292 296 292
query95 348 253 244 244
query96 615 288 292 288
query97 3286 3137 3150 3137
query98 215 199 198 198
query99 1651 1321 1323 1321
Total cold run time: 300562 ms
Total hot run time: 191866 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1797530637
##########
regression-test/suites/query_p0/sql_functions/string_functions/test_trim_in.groovy:
##########
@@ -0,0 +1,153 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_trim_in") {
+ // this table has nothing todo. just make it eaiser to generate query
+ sql " drop table if exists hits_two_args "
+ sql """ create table hits_two_args(
+ nothing boolean
+ )
+ properties("replication_num" = "1");
+ """
+ sql "insert into hits_two_args values(true);"
+
+ sql " drop table if exists test_trim_in"
+ sql """
+ create table test_trim_in (
+ k0 int,
+ a varchar not null,
+ b varchar null
+ )
+ DISTRIBUTED BY HASH(k0)
+ PROPERTIES
+ (
+ "replication_num" = "1"
+ );
+ """
+ /// consts. most by BE-UT
+ order_qt_const_nullable "select trim_in(NULL,NULL) from test_trim_in"
+ order_qt_partial_const_nullable "select trim_in(NULL, b) from test_trim_in"
+ order_qt_const_not_nullable "select trim_in(a,b) from test_trim_in"
+ order_qt_const_other_nullable "select trim_in('x',b) from test_trim_in"
+ order_qt_const_other_not_nullable "select trim_in('x',a) from test_trim_in"
+ order_qt_const_partial_nullable_no_null "select trim_in('abc', 'b')"
+ //[INVALID_ARGUMENT]Argument at index 1 for function trim must be constant
+ //order_qt_const_nullable_no_null "select trim_in(nullable('abc'),nullable('中文'))"
+ //order_qt_const_partial_nullable_no_null "select trim_in('abc',nullable('a'))"
+ //order_qt_const_nullable_no_null_multirows "select trim_in(nullable('abc'), nullable('a'))"
+
+
+
+ /// folding
+ def re_fe
+ def re_be
+ def re_no_fold
+ def check_three_ways = { test_sql ->
+ re_fe = order_sql "select/*+SET_VAR(enable_fold_constant_by_be=false)*/ ${test_sql}"
+ re_be = order_sql "select/*+SET_VAR(enable_fold_constant_by_be=true)*/ ${test_sql}"
+ re_no_fold = order_sql "select/*+SET_VAR(debug_skip_fold_constant=true)*/ ${test_sql}"
+ logger.info("check on sql ${test_sql}")
+ assertEquals(re_fe, re_be)
+ assertEquals(re_fe, re_no_fold)
+ }
+
+ check_three_ways "trim_in(' hello world ', ' ld')"
+ check_three_ways "ltrim_in(' hello world ', ' ld')"
+ check_three_ways "rtrim_in(' hello world ', ' ld')"
+ check_three_ways "trim_in(' hello world ', ' ehlowrd')"
+ check_three_ways "ltrim_in(' hello world ', ' ehlowrd')"
+ check_three_ways "rtrim_in(' hello world ', ' ehlowrd')"
+ check_three_ways "trim_in(' hello world ', '')"
+ check_three_ways "ltrim_in(' hello world ', '')"
+ check_three_ways "rtrim_in(' hello world ', '')"
+ check_three_ways "trim_in(' hello world ', ' ')"
+ check_three_ways "ltrim_in(' hello world ', ' ')"
+ check_three_ways "rtrim_in(' hello world ', ' ')"
+
+ order_qt_1 "SELECT ltrim_in('');"
+ order_qt_2 "SELECT ltrim_in(' ');"
+ order_qt_3 "SELECT ltrim_in(' hello ');"
+ order_qt_4 "SELECT ltrim_in(' hello');"
+ order_qt_5 "SELECT ltrim_in('hello ');"
+ order_qt_6 "SELECT ltrim_in(' hello world ');"
+ order_qt_7 "SELECT ltrim_in(CAST('' AS CHAR(20)));"
+ order_qt_8 "SELECT ltrim_in(CAST(' hello ' AS CHAR(9)));"
+ order_qt_9 "SELECT ltrim_in(CAST(' hello' AS CHAR(7)));"
+ order_qt_10 "SELECT ltrim_in(CAST('hello ' AS CHAR(7)));"
+ order_qt_11 "SELECT ltrim_in(CAST(' hello world ' AS CHAR(13)));"
+ order_qt_12 "SELECT rtrim_in('');"
+ order_qt_13 "SELECT rtrim_in(' ');"
+ order_qt_14 "SELECT rtrim_in(' hello ');"
+ order_qt_15 "SELECT rtrim_in(' hello');"
+ order_qt_16 "SELECT rtrim_in('hello ');"
+ order_qt_17 "SELECT rtrim_in(' hello world ');"
+ order_qt_18 "SELECT rtrim_in(CAST('' AS CHAR(20)));"
+ order_qt_19 "SELECT rtrim_in(CAST(' hello ' AS CHAR(9)));"
+ order_qt_20 "SELECT rtrim_in(CAST(' hello' AS CHAR(7)));"
+ order_qt_21 "SELECT rtrim_in(CAST('hello ' AS CHAR(7)));"
+ order_qt_22 "SELECT rtrim_in(CAST(' hello world ' AS CHAR(13)));"
+ order_qt_23 "SELECT ltrim_in('', '');"
+ order_qt_24 "SELECT ltrim_in(' ', '');"
+ order_qt_25 "SELECT ltrim_in(' hello ', '');"
+ order_qt_26 "SELECT ltrim_in(' hello ', ' ');"
+ order_qt_27 "SELECT ltrim_in(' hello ', ' ');"
+ order_qt_28 "SELECT ltrim_in(' hello ', 'he ');"
+ order_qt_29 "SELECT ltrim_in(' hello', ' ');"
+ order_qt_30 "SELECT ltrim_in(' hello', 'e h');"
+ order_qt_31 "SELECT ltrim_in('hello ', 'l');"
+ order_qt_32 "SELECT ltrim_in(' hello world ', ' ');"
+ order_qt_33 "SELECT ltrim_in(' hello world ', ' eh');"
+ order_qt_34 "SELECT ltrim_in(' hello world ', ' ehlowrd');"
+ order_qt_35 "SELECT ltrim_in(' hello world ', ' x');"
+ order_qt_36 "SELECT ltrim_in(CAST('' AS CHAR(1)), '');"
+ order_qt_37 "SELECT ltrim_in(CAST(' ' AS CHAR(3)), '');"
+ order_qt_38 "SELECT ltrim_in(CAST(' hello ' AS CHAR(9)), '');"
+ order_qt_39 "SELECT ltrim_in(CAST(' hello ' AS CHAR(9)), ' ');"
+ order_qt_40 "SELECT ltrim_in(CAST(' hello ' AS CHAR(9)), 'he ');"
+ order_qt_41 "SELECT ltrim_in(CAST(' hello' AS CHAR(7)), ' ');"
+ order_qt_42 "SELECT ltrim_in(CAST(' hello' AS CHAR(7)), 'e h');"
+ order_qt_43 "SELECT ltrim_in(CAST('hello ' AS CHAR(7)), 'l');"
+ order_qt_44 "SELECT ltrim_in(CAST(' hello world ' AS CHAR(13)), ' ');"
+ order_qt_45 "SELECT ltrim_in(CAST(' hello world ' AS CHAR(13)), ' eh');"
+ order_qt_46 "SELECT ltrim_in(CAST(' hello world ' AS CHAR(13)), ' ehlowrd');"
+ order_qt_47 "SELECT ltrim_in(CAST(' hello world ' AS CHAR(13)), ' x');"
+ order_qt_48 "SELECT rtrim_in('', '');"
+ order_qt_49 "SELECT rtrim_in(' ', '');"
+ order_qt_50 "SELECT rtrim_in(' hello ', '');"
+ order_qt_51 "SELECT rtrim_in(' hello ', ' ');"
+ order_qt_52 "SELECT rtrim_in(' hello ', 'lo ');"
+ order_qt_53 "SELECT rtrim_in('hello ', ' ');"
+ order_qt_54 "SELECT rtrim_in('hello ', 'l o');"
+ order_qt_55 "SELECT rtrim_in('hello ', 'l');"
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2415943590
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "zhiqiang-hhhh (via GitHub)" <gi...@apache.org>.
zhiqiang-hhhh commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2415548364
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2415612532
<details>
<summary>TPC-H: <b>Total hot run time: 41520 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 90febeb9c3d1eb0ab8100cf4ac008b9e55dc8412, data reload: false
------ Round 1 ----------------------------------
q1 17577 7542 7342 7342
q2 2050 169 167 167
q3 10557 1160 1191 1160
q4 10222 816 862 816
q5 7776 3116 3098 3098
q6 236 155 155 155
q7 1020 610 603 603
q8 9365 1981 1974 1974
q9 6680 6498 6474 6474
q10 7104 2472 2429 2429
q11 445 249 244 244
q12 413 221 220 220
q13 17763 3037 3012 3012
q14 251 222 214 214
q15 588 537 508 508
q16 630 587 587 587
q17 995 687 483 483
q18 7398 6728 6663 6663
q19 1354 963 1041 963
q20 482 192 185 185
q21 4054 3289 3218 3218
q22 1129 1015 1005 1005
Total cold run time: 108089 ms
Total hot run time: 41520 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7277 7249 7244 7244
q2 326 233 223 223
q3 2948 2742 2823 2742
q4 1963 1741 1692 1692
q5 5471 5509 5565 5509
q6 227 146 145 145
q7 2186 1715 1699 1699
q8 3272 3406 3427 3406
q9 8634 8659 8581 8581
q10 3526 3471 3477 3471
q11 572 485 490 485
q12 795 604 600 600
q13 8969 3051 3013 3013
q14 291 264 269 264
q15 557 516 508 508
q16 680 643 644 643
q17 1842 1605 1568 1568
q18 7870 7649 7622 7622
q19 1695 1568 1492 1492
q20 2084 1795 1850 1795
q21 5373 5358 5241 5241
q22 1112 1056 1029 1029
Total cold run time: 67670 ms
Total hot run time: 58972 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2415622871
<details>
<summary>TPC-DS: <b>Total hot run time: 192329 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 90febeb9c3d1eb0ab8100cf4ac008b9e55dc8412, data reload: false
query1 991 392 378 378
query2 6539 2122 2078 2078
query3 6791 211 223 211
query4 34500 23515 23604 23515
query5 4392 464 454 454
query6 263 165 166 165
query7 4597 293 296 293
query8 301 240 241 240
query9 9788 2751 2748 2748
query10 498 273 273 273
query11 18128 15279 15395 15279
query12 151 105 103 103
query13 1663 450 426 426
query14 10328 7576 7634 7576
query15 311 170 170 170
query16 8081 480 475 475
query17 1799 568 581 568
query18 2144 302 304 302
query19 370 154 153 153
query20 121 112 107 107
query21 220 104 106 104
query22 4573 4359 4135 4135
query23 35044 33943 34093 33943
query24 11116 2821 2784 2784
query25 657 415 404 404
query26 1380 166 165 165
query27 2857 286 293 286
query28 7922 2472 2423 2423
query29 906 444 424 424
query30 325 157 152 152
query31 1040 790 828 790
query32 100 56 57 56
query33 775 306 308 306
query34 938 518 532 518
query35 921 752 740 740
query36 1103 937 959 937
query37 164 93 93 93
query38 4085 3912 3915 3912
query39 1482 1422 1426 1422
query40 292 99 102 99
query41 49 46 46 46
query42 118 101 99 99
query43 538 505 492 492
query44 1333 816 819 816
query45 199 170 170 170
query46 1133 704 702 702
query47 1925 1839 1823 1823
query48 425 332 341 332
query49 1169 445 436 436
query50 841 385 397 385
query51 7076 6968 7055 6968
query52 101 89 90 89
query53 263 181 185 181
query54 1285 447 462 447
query55 83 78 81 78
query56 285 283 290 283
query57 1308 1169 1159 1159
query58 257 281 252 252
query59 3191 3076 2896 2896
query60 310 278 272 272
query61 112 100 100 100
query62 875 685 655 655
query63 225 184 190 184
query64 5269 675 609 609
query65 3327 3260 3230 3230
query66 1448 314 307 307
query67 16058 15779 15626 15626
query68 4937 570 592 570
query69 486 294 313 294
query70 1218 1126 1136 1126
query71 354 276 276 276
query72 7057 4080 4029 4029
query73 802 359 361 359
query74 10224 9017 9008 9008
query75 3427 2679 2682 2679
query76 2900 931 953 931
query77 440 300 294 294
query78 10617 9694 9544 9544
query79 2217 592 625 592
query80 1097 473 452 452
query81 583 244 246 244
query82 1090 143 135 135
query83 250 134 142 134
query84 261 67 71 67
query85 1240 320 289 289
query86 350 304 311 304
query87 4532 4445 4395 4395
query88 3719 2233 2186 2186
query89 402 289 299 289
query90 1951 189 190 189
query91 142 102 109 102
query92 68 50 49 49
query93 1140 551 541 541
query94 965 281 268 268
query95 348 251 252 251
query96 618 284 281 281
query97 3253 3110 3126 3110
query98 217 212 196 196
query99 1662 1313 1301 1301
Total cold run time: 305426 ms
Total hot run time: 192329 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1796700303
##########
regression-test/suites/query_p0/sql_functions/string_functions/test_trim_in.groovy:
##########
@@ -0,0 +1,153 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_trim_in") {
+ // this table has nothing todo. just make it eaiser to generate query
+ sql " drop table if exists hits_two_args "
+ sql """ create table hits_two_args(
+ nothing boolean
+ )
+ properties("replication_num" = "1");
+ """
+ sql "insert into hits_two_args values(true);"
+
+ sql " drop table if exists test_trim_in"
+ sql """
+ create table test_trim_in (
+ k0 int,
+ a varchar not null,
+ b varchar null
+ )
+ DISTRIBUTED BY HASH(k0)
+ PROPERTIES
+ (
+ "replication_num" = "1"
+ );
+ """
+ /// consts. most by BE-UT
+ order_qt_const_nullable "select trim_in(NULL,NULL) from test_trim_in"
+ order_qt_partial_const_nullable "select trim_in(NULL, b) from test_trim_in"
+ order_qt_const_not_nullable "select trim_in(a,b) from test_trim_in"
+ order_qt_const_other_nullable "select trim_in('x',b) from test_trim_in"
+ order_qt_const_other_not_nullable "select trim_in('x',a) from test_trim_in"
+ order_qt_const_partial_nullable_no_null "select trim_in('abc', 'b')"
+ //[INVALID_ARGUMENT]Argument at index 1 for function trim must be constant
+ //order_qt_const_nullable_no_null "select trim_in(nullable('abc'),nullable('中文'))"
+ //order_qt_const_partial_nullable_no_null "select trim_in('abc',nullable('a'))"
+ //order_qt_const_nullable_no_null_multirows "select trim_in(nullable('abc'), nullable('a'))"
+
+
+
+ /// folding
+ def re_fe
+ def re_be
+ def re_no_fold
+ def check_three_ways = { test_sql ->
+ re_fe = order_sql "select/*+SET_VAR(enable_fold_constant_by_be=false)*/ ${test_sql}"
+ re_be = order_sql "select/*+SET_VAR(enable_fold_constant_by_be=true)*/ ${test_sql}"
+ re_no_fold = order_sql "select/*+SET_VAR(debug_skip_fold_constant=true)*/ ${test_sql}"
+ logger.info("check on sql ${test_sql}")
+ assertEquals(re_fe, re_be)
+ assertEquals(re_fe, re_no_fold)
+ }
+
+ check_three_ways "trim_in(' hello world ', ' ld')"
+ check_three_ways "ltrim_in(' hello world ', ' ld')"
+ check_three_ways "rtrim_in(' hello world ', ' ld')"
+ check_three_ways "trim_in(' hello world ', ' ehlowrd')"
+ check_three_ways "ltrim_in(' hello world ', ' ehlowrd')"
+ check_three_ways "rtrim_in(' hello world ', ' ehlowrd')"
+ check_three_ways "trim_in(' hello world ', '')"
+ check_three_ways "ltrim_in(' hello world ', '')"
+ check_three_ways "rtrim_in(' hello world ', '')"
+ check_three_ways "trim_in(' hello world ', ' ')"
+ check_three_ways "ltrim_in(' hello world ', ' ')"
+ check_three_ways "rtrim_in(' hello world ', ' ')"
+
+ order_qt_1 "SELECT ltrim_in('');"
+ order_qt_2 "SELECT ltrim_in(' ');"
+ order_qt_3 "SELECT ltrim_in(' hello ');"
+ order_qt_4 "SELECT ltrim_in(' hello');"
+ order_qt_5 "SELECT ltrim_in('hello ');"
+ order_qt_6 "SELECT ltrim_in(' hello world ');"
+ order_qt_7 "SELECT ltrim_in(CAST('' AS CHAR(20)));"
+ order_qt_8 "SELECT ltrim_in(CAST(' hello ' AS CHAR(9)));"
+ order_qt_9 "SELECT ltrim_in(CAST(' hello' AS CHAR(7)));"
+ order_qt_10 "SELECT ltrim_in(CAST('hello ' AS CHAR(7)));"
+ order_qt_11 "SELECT ltrim_in(CAST(' hello world ' AS CHAR(13)));"
+ order_qt_12 "SELECT rtrim_in('');"
+ order_qt_13 "SELECT rtrim_in(' ');"
+ order_qt_14 "SELECT rtrim_in(' hello ');"
+ order_qt_15 "SELECT rtrim_in(' hello');"
+ order_qt_16 "SELECT rtrim_in('hello ');"
+ order_qt_17 "SELECT rtrim_in(' hello world ');"
+ order_qt_18 "SELECT rtrim_in(CAST('' AS CHAR(20)));"
+ order_qt_19 "SELECT rtrim_in(CAST(' hello ' AS CHAR(9)));"
+ order_qt_20 "SELECT rtrim_in(CAST(' hello' AS CHAR(7)));"
+ order_qt_21 "SELECT rtrim_in(CAST('hello ' AS CHAR(7)));"
+ order_qt_22 "SELECT rtrim_in(CAST(' hello world ' AS CHAR(13)));"
+ order_qt_23 "SELECT ltrim_in('', '');"
+ order_qt_24 "SELECT ltrim_in(' ', '');"
+ order_qt_25 "SELECT ltrim_in(' hello ', '');"
+ order_qt_26 "SELECT ltrim_in(' hello ', ' ');"
+ order_qt_27 "SELECT ltrim_in(' hello ', ' ');"
+ order_qt_28 "SELECT ltrim_in(' hello ', 'he ');"
+ order_qt_29 "SELECT ltrim_in(' hello', ' ');"
+ order_qt_30 "SELECT ltrim_in(' hello', 'e h');"
+ order_qt_31 "SELECT ltrim_in('hello ', 'l');"
+ order_qt_32 "SELECT ltrim_in(' hello world ', ' ');"
+ order_qt_33 "SELECT ltrim_in(' hello world ', ' eh');"
+ order_qt_34 "SELECT ltrim_in(' hello world ', ' ehlowrd');"
+ order_qt_35 "SELECT ltrim_in(' hello world ', ' x');"
+ order_qt_36 "SELECT ltrim_in(CAST('' AS CHAR(1)), '');"
+ order_qt_37 "SELECT ltrim_in(CAST(' ' AS CHAR(3)), '');"
+ order_qt_38 "SELECT ltrim_in(CAST(' hello ' AS CHAR(9)), '');"
+ order_qt_39 "SELECT ltrim_in(CAST(' hello ' AS CHAR(9)), ' ');"
+ order_qt_40 "SELECT ltrim_in(CAST(' hello ' AS CHAR(9)), 'he ');"
+ order_qt_41 "SELECT ltrim_in(CAST(' hello' AS CHAR(7)), ' ');"
+ order_qt_42 "SELECT ltrim_in(CAST(' hello' AS CHAR(7)), 'e h');"
+ order_qt_43 "SELECT ltrim_in(CAST('hello ' AS CHAR(7)), 'l');"
+ order_qt_44 "SELECT ltrim_in(CAST(' hello world ' AS CHAR(13)), ' ');"
+ order_qt_45 "SELECT ltrim_in(CAST(' hello world ' AS CHAR(13)), ' eh');"
+ order_qt_46 "SELECT ltrim_in(CAST(' hello world ' AS CHAR(13)), ' ehlowrd');"
+ order_qt_47 "SELECT ltrim_in(CAST(' hello world ' AS CHAR(13)), ' x');"
+ order_qt_48 "SELECT rtrim_in('', '');"
+ order_qt_49 "SELECT rtrim_in(' ', '');"
+ order_qt_50 "SELECT rtrim_in(' hello ', '');"
+ order_qt_51 "SELECT rtrim_in(' hello ', ' ');"
+ order_qt_52 "SELECT rtrim_in(' hello ', 'lo ');"
+ order_qt_53 "SELECT rtrim_in('hello ', ' ');"
+ order_qt_54 "SELECT rtrim_in('hello ', 'l o');"
+ order_qt_55 "SELECT rtrim_in('hello ', 'l');"
Review Comment:
add more test
1. create a table and then insert the values to a table, and use select func(col) from table
2. the column in the table, could be a nullable column and not null column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1796704190
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -535,6 +545,42 @@ struct TrimUtil {
return Status::OK();
}
};
+template <bool is_ltrim_in, bool is_rtrim_in, bool trim_single>
+struct TrimInUtil {
+ static Status vector(const ColumnString::Chars& str_data,
Review Comment:
为什么不直接使用column string，而是把chars和offsets 分开
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2415941677
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2405343214
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2415609346
TeamCity be ut coverage result:
Function Coverage: 37.45% (9709/25922)
Line Coverage: 28.72% (80624/280699)
Region Coverage: 28.16% (41701/148101)
Branch Coverage: 24.74% (21206/85720)
Coverage Report: http://coverage.selectdb-in.cc/coverage/90febeb9c3d1eb0ab8100cf4ac008b9e55dc8412_90febeb9c3d1eb0ab8100cf4ac008b9e55dc8412/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "liujiwen-up (via GitHub)" <gi...@apache.org>.
liujiwen-up commented on code in PR #41681:
URL: https://github.com/apache/doris/pull/41681#discussion_r1800408496
##########
be/src/vec/functions/function_string.cpp:
##########
@@ -535,6 +545,42 @@ struct TrimUtil {
return Status::OK();
}
};
+template <bool is_ltrim_in, bool is_rtrim_in, bool trim_single>
+struct TrimInUtil {
+ static Status vector(const ColumnString::Chars& str_data,
+ const ColumnString::Offsets& str_offsets, const StringRef& remove_str,
+ ColumnString::Chars& res_data, ColumnString::Offsets& res_offsets) {
+ const size_t offset_size = str_offsets.size();
+ res_offsets.resize(offset_size);
+ res_data.reserve(str_data.size());
+ std::bitset<256> char_lookup;
Review Comment:
Currently, there are two processing logics.
1. The `simd::VStringFunctions::is_ascii `method is used to judge that when the string is all ascii, bitset<128> will be used for processing. The character range of the standard ASCII table is 0 to 127. Bitset<128> has exactly 128 bits, which is enough to represent all standard ASCII characters.
2. When the string is not all standard ascii, the utf-8 logic will be used for processing.It is especially important to note that when trimming on the right, according to the rules of UTF-8, the format of the UTF-8 trailing byte is 10xxxxxx. Use `(*prev_char_pos & 0xC0) == 0x80` to determine whether the current byte is a trailing byte. When a byte that is not a trailing byte is found, this byte is the starting byte of the current character.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2415649664
Please add doc in https://github.com/apache/doris-website @liujiwen-up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2418817676
<details>
<summary>TPC-H: <b>Total hot run time: 41202 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c741d6d007719db8af2bfdb3cbb9af8f2a05f531, data reload: false
------ Round 1 ----------------------------------
q1 17585 7355 7239 7239
q2 2040 272 268 268
q3 12070 1082 1125 1082
q4 10568 832 849 832
q5 7759 3078 3039 3039
q6 240 151 150 150
q7 1020 619 601 601
q8 9425 1939 1994 1939
q9 7502 6447 6474 6447
q10 7030 2413 2415 2413
q11 435 240 241 240
q12 411 222 224 222
q13 17769 3061 3004 3004
q14 256 213 212 212
q15 570 522 531 522
q16 654 598 581 581
q17 974 540 563 540
q18 7203 6804 6627 6627
q19 1343 1008 1091 1008
q20 456 183 181 181
q21 4030 3074 3205 3074
q22 1072 1015 981 981
Total cold run time: 110412 ms
Total hot run time: 41202 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7285 7238 7324 7238
q2 326 236 238 236
q3 3043 2958 2940 2940
q4 2042 1860 1795 1795
q5 5808 5807 5800 5800
q6 246 147 148 147
q7 2247 1844 1834 1834
q8 3402 3519 3489 3489
q9 8988 8918 8924 8918
q10 3609 3596 3564 3564
q11 613 501 490 490
q12 868 666 641 641
q13 11305 3229 3190 3190
q14 309 273 269 269
q15 566 521 519 519
q16 691 666 649 649
q17 1884 1597 1600 1597
q18 8426 7633 7610 7610
q19 1727 1558 1458 1458
q20 2155 1884 1885 1884
q21 5613 5584 5366 5366
q22 1117 1054 1067 1054
Total cold run time: 72270 ms
Total hot run time: 60688 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](functions) impl scalar functions trim_in、ltrim_in and rtrim_in [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41681:
URL: https://github.com/apache/doris/pull/41681#issuecomment-2418856929
<details>
<summary>ClickBench: <b>Total hot run time: 33.81 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c741d6d007719db8af2bfdb3cbb9af8f2a05f531, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.03 0.02
query3 0.23 0.06 0.07
query4 1.65 0.10 0.09
query5 0.49 0.50 0.51
query6 1.13 0.72 0.73
query7 0.02 0.02 0.02
query8 0.03 0.03 0.02
query9 0.55 0.51 0.50
query10 0.56 0.55 0.56
query11 0.13 0.11 0.11
query12 0.14 0.11 0.10
query13 0.61 0.59 0.60
query14 2.72 2.83 2.71
query15 0.91 0.83 0.84
query16 0.38 0.37 0.40
query17 1.06 1.06 1.06
query18 0.20 0.20 0.20
query19 1.93 1.89 1.95
query20 0.01 0.00 0.01
query21 15.36 0.60 0.57
query22 2.46 2.00 2.27
query23 17.08 1.03 0.83
query24 2.74 1.80 2.37
query25 0.19 0.22 0.10
query26 0.62 0.14 0.14
query27 0.04 0.04 0.05
query28 9.43 1.10 1.06
query29 12.54 3.28 3.29
query30 0.24 0.06 0.06
query31 2.87 0.39 0.40
query32 3.28 0.47 0.46
query33 2.99 3.11 2.99
query34 17.14 4.49 4.47
query35 4.50 4.48 4.47
query36 0.65 0.49 0.48
query37 0.08 0.06 0.06
query38 0.04 0.04 0.03
query39 0.03 0.03 0.02
query40 0.16 0.12 0.12
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 105.46 s
Total hot run time: 33.81 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org