You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zhangstar333 (via GitHub)" <gi...@apache.org> on 2025/04/21 09:26:18 UTC
[PR] [improve](function) support regexp_replace function with ignore flag options [doris]
zhangstar333 opened a new pull request, #50245:
URL: https://github.com/apache/doris/pull/50245
### What problem does this PR solve?
Problem Summary:
in this PR https://github.com/apache/doris/pull/50124, add an option **ignore_replace_escape**.
so now support
`select regexp_replace('abc', 'b', '\}', "IGNORE_INVALID_ESCAPE");`
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2826324394
<details>
<summary>TPC-H: <b>Total hot run time: 33792 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 03f436ed253e072e6f4ee8759493d0374dd8cffb, data reload: false
------ Round 1 ----------------------------------
q1 25881 5107 4982 4982
q2 2074 269 179 179
q3 10439 1244 680 680
q4 10262 996 540 540
q5 8254 2473 2332 2332
q6 188 164 133 133
q7 914 746 611 611
q8 9324 1265 1078 1078
q9 6847 5158 5167 5158
q10 6801 2308 1852 1852
q11 475 286 286 286
q12 350 353 214 214
q13 17757 3659 3101 3101
q14 235 221 208 208
q15 545 491 482 482
q16 449 443 402 402
q17 579 858 353 353
q18 7614 7198 7019 7019
q19 1337 948 564 564
q20 345 325 218 218
q21 3912 3329 2438 2438
q22 1043 1000 962 962
Total cold run time: 115625 ms
Total hot run time: 33792 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5109 5043 5082 5043
q2 240 331 231 231
q3 2177 2637 2320 2320
q4 1408 1790 1383 1383
q5 4615 4466 4407 4407
q6 203 165 124 124
q7 1957 1901 1722 1722
q8 2627 2533 2636 2533
q9 7201 7214 7140 7140
q10 3007 3197 2746 2746
q11 578 511 512 511
q12 711 787 642 642
q13 3454 3877 3345 3345
q14 273 288 265 265
q15 519 475 473 473
q16 480 515 481 481
q17 1166 1560 1328 1328
q18 7824 7635 7317 7317
q19 787 776 856 776
q20 1964 2041 1872 1872
q21 5233 4619 4626 4619
q22 1083 1051 1001 1001
Total cold run time: 52616 ms
Total hot run time: 50279 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2819935442
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2819054489
<details>
<summary>TPC-DS: <b>Total hot run time: 192890 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a52c323e58980ddf32fb800cc6885043380739b2, data reload: false
query1 1437 1076 1062 1062
query2 6104 1815 1822 1815
query3 10985 4698 4569 4569
query4 54589 25843 23403 23403
query5 5014 506 466 466
query6 336 222 210 210
query7 4878 501 290 290
query8 322 249 231 231
query9 5707 2573 2570 2570
query10 429 327 267 267
query11 15015 15036 15117 15036
query12 160 112 102 102
query13 1038 517 405 405
query14 10324 6448 6346 6346
query15 207 201 183 183
query16 7218 670 530 530
query17 1113 770 603 603
query18 1703 427 330 330
query19 202 201 177 177
query20 176 130 121 121
query21 211 131 99 99
query22 4305 4420 4332 4332
query23 34253 33291 33430 33291
query24 6570 2468 2428 2428
query25 482 461 443 443
query26 695 276 159 159
query27 2218 525 354 354
query28 3041 2163 2150 2150
query29 596 570 420 420
query30 268 229 184 184
query31 876 848 775 775
query32 76 65 66 65
query33 449 361 319 319
query34 767 878 522 522
query35 806 849 746 746
query36 946 1011 927 927
query37 111 111 81 81
query38 4230 4231 4154 4154
query39 1516 1419 1466 1419
query40 219 116 108 108
query41 54 53 53 53
query42 120 116 103 103
query43 503 519 495 495
query44 1369 802 804 802
query45 182 189 174 174
query46 863 1037 656 656
query47 1824 1856 1797 1797
query48 384 409 317 317
query49 702 516 425 425
query50 685 725 435 435
query51 4200 4307 4158 4158
query52 112 105 101 101
query53 241 260 188 188
query54 583 574 516 516
query55 83 84 82 82
query56 316 305 310 305
query57 1219 1195 1092 1092
query58 266 268 274 268
query59 2678 2818 2613 2613
query60 359 366 317 317
query61 132 130 130 130
query62 755 727 720 720
query63 234 194 194 194
query64 1816 1078 698 698
query65 4422 4330 4277 4277
query66 703 400 308 308
query67 15978 15829 15319 15319
query68 7354 895 513 513
query69 529 295 319 295
query70 1215 1113 1110 1110
query71 494 312 295 295
query72 5784 4784 4855 4784
query73 1456 644 425 425
query74 9009 9244 8729 8729
query75 3757 3213 2742 2742
query76 4251 1222 802 802
query77 619 391 289 289
query78 9999 10073 9292 9292
query79 5912 809 541 541
query80 648 510 438 438
query81 537 266 223 223
query82 670 138 94 94
query83 352 248 230 230
query84 290 102 82 82
query85 786 358 314 314
query86 366 303 285 285
query87 4442 4457 4253 4253
query88 2928 2191 2201 2191
query89 457 311 284 284
query90 1921 219 225 219
query91 159 143 108 108
query92 71 65 57 57
query93 2964 935 575 575
query94 689 420 312 312
query95 364 297 293 293
query96 498 563 278 278
query97 3175 3238 3112 3112
query98 220 205 202 202
query99 1475 1382 1288 1288
Total cold run time: 303766 ms
Total hot run time: 192890 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2826256167
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2820008311
<details>
<summary>ClickBench: <b>Total hot run time: 29.44 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0532e1409c8a06da08431119687bc36b125c1c5b, data reload: false
query1 0.04 0.04 0.03
query2 0.12 0.10 0.10
query3 0.26 0.19 0.19
query4 1.60 0.19 0.19
query5 0.60 0.58 0.59
query6 1.19 0.71 0.72
query7 0.02 0.02 0.01
query8 0.05 0.04 0.03
query9 0.58 0.53 0.51
query10 0.57 0.57 0.56
query11 0.16 0.10 0.11
query12 0.14 0.11 0.12
query13 0.61 0.60 0.60
query14 1.14 1.21 1.20
query15 0.87 0.85 0.88
query16 0.38 0.39 0.38
query17 1.02 1.05 1.04
query18 0.21 0.19 0.20
query19 1.89 1.83 1.90
query20 0.01 0.01 0.01
query21 15.48 0.88 0.54
query22 0.77 1.14 0.70
query23 14.95 1.39 0.66
query24 6.70 1.83 0.42
query25 0.43 0.18 0.12
query26 0.61 0.16 0.14
query27 0.05 0.05 0.05
query28 9.08 0.91 0.44
query29 12.54 3.89 3.30
query30 0.25 0.10 0.08
query31 2.81 0.60 0.38
query32 3.23 0.56 0.47
query33 3.02 3.13 3.04
query34 15.70 5.15 4.53
query35 4.54 4.53 4.53
query36 0.67 0.49 0.48
query37 0.08 0.07 0.06
query38 0.06 0.04 0.04
query39 0.03 0.03 0.03
query40 0.17 0.14 0.13
query41 0.09 0.03 0.03
query42 0.04 0.02 0.03
query43 0.04 0.03 0.04
Total cold run time: 102.8 s
Total hot run time: 29.44 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2823189682
# BE UT Coverage Report
Increment line coverage `70.69% (82/116)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/150af8994ad2ffe2aa230b879eca56b2b9a54c3c_150af8994ad2ffe2aa230b879eca56b2b9a54c3c/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/150af8994ad2ffe2aa230b879eca56b2b9a54c3c_150af8994ad2ffe2aa230b879eca56b2b9a54c3c/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.55% (14543/27159) |
| Line Coverage | 42.38% (126082/297505) |
| Region Coverage | 41.19% (64426/156417) |
| Branch Coverage | 35.77% (32387/90550) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2819935495
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2826349557
<details>
<summary>ClickBench: <b>Total hot run time: 29.8 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 03f436ed253e072e6f4ee8759493d0374dd8cffb, data reload: false
query1 0.04 0.04 0.03
query2 0.11 0.10 0.11
query3 0.26 0.19 0.21
query4 1.59 0.19 0.20
query5 0.58 0.59 0.60
query6 1.17 0.72 0.72
query7 0.02 0.02 0.01
query8 0.04 0.04 0.04
query9 0.59 0.54 0.51
query10 0.56 0.56 0.57
query11 0.15 0.11 0.11
query12 0.15 0.11 0.12
query13 0.61 0.61 0.60
query14 1.16 1.18 1.20
query15 0.89 0.85 0.86
query16 0.38 0.38 0.38
query17 1.01 1.01 1.01
query18 0.21 0.20 0.19
query19 1.94 1.80 1.76
query20 0.01 0.02 0.01
query21 15.44 1.04 0.55
query22 0.78 1.28 0.71
query23 14.80 1.44 0.62
query24 6.83 1.26 0.99
query25 0.45 0.14 0.08
query26 0.58 0.17 0.14
query27 0.05 0.06 0.05
query28 9.75 0.93 0.45
query29 12.56 3.94 3.29
query30 0.26 0.08 0.06
query31 2.84 0.57 0.38
query32 3.23 0.55 0.47
query33 3.03 3.04 3.10
query34 15.85 5.17 4.49
query35 4.49 4.56 4.54
query36 0.66 0.48 0.48
query37 0.09 0.06 0.06
query38 0.05 0.03 0.04
query39 0.02 0.02 0.02
query40 0.17 0.14 0.14
query41 0.08 0.02 0.02
query42 0.03 0.03 0.02
query43 0.04 0.03 0.03
Total cold run time: 103.55 s
Total hot run time: 29.8 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2840704462
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #50245:
URL: https://github.com/apache/doris/pull/50245#discussion_r2052220438
##########
be/src/vec/functions/function_regexp.cpp:
##########
@@ -52,12 +51,129 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct ThreeParamTypes {
+ static DataTypes get_variadic_argument_types() {
+ return {std::make_shared<DataTypeString>(), std::make_shared<DataTypeString>(),
+ std::make_shared<DataTypeString>()};
+ }
+};
+
+struct FourParamTypes {
+ static DataTypes get_variadic_argument_types() {
+ return {std::make_shared<DataTypeString>(), std::make_shared<DataTypeString>(),
+ std::make_shared<DataTypeString>(), std::make_shared<DataTypeString>()};
+ }
+};
+
+// template FunctionRegexpFunctionality is used for regexp_replace/regexp_replace_one
+template <typename Impl, typename ParamTypes>
+class FunctionRegexpReplace : public IFunction {
+public:
+ static constexpr auto name = Impl::name;
+
+ static FunctionPtr create() { return std::make_shared<FunctionRegexpReplace>(); }
+
+ String get_name() const override { return name; }
+
+ size_t get_number_of_arguments() const override {
+ return get_variadic_argument_types_impl().size();
+ }
+
+ bool is_variadic() const override { return true; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ return make_nullable(std::make_shared<DataTypeString>());
+ }
+
+ DataTypes get_variadic_argument_types_impl() const override {
+ return ParamTypes::get_variadic_argument_types();
+ }
+
+ Status open(FunctionContext* context, FunctionContext::FunctionStateScope scope) override {
+ if (scope == FunctionContext::THREAD_LOCAL) {
+ if (context->is_col_constant(1)) {
+ DCHECK(!context->get_function_state(scope));
+ const auto pattern_col = context->get_constant_col(1)->column_ptr;
+ const auto& pattern = pattern_col->get_data_at(0);
+ if (pattern.size == 0) {
+ return Status::OK();
+ }
+
+ std::string error_str;
+ std::unique_ptr<re2::RE2> scoped_re;
+ StringRef options_value;
+ if (context->get_num_args() == 4) {
+ DCHECK(context->is_col_constant(3));
+ const auto options_col = context->get_constant_col(3)->column_ptr;
+ options_value = options_col->get_data_at(0);
+ }
+
+ bool st = StringFunctions::compile_regex(pattern, &error_str, StringRef(),
+ options_value, scoped_re);
+ if (!st) {
+ context->set_error(error_str.c_str());
+ return Status::InvalidArgument(error_str);
+ }
+ std::shared_ptr<re2::RE2> re(scoped_re.release());
+ context->set_function_state(scope, re);
+ }
+ }
+ return Status::OK();
+ }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ uint32_t result, size_t input_rows_count) const override {
+ size_t argument_size = arguments.size();
+
+ auto result_null_map = ColumnUInt8::create(input_rows_count, 0);
+ auto result_data_column = ColumnString::create();
+ auto& result_data = result_data_column->get_chars();
+ auto& result_offset = result_data_column->get_offsets();
+ result_offset.resize(input_rows_count);
+
+ bool col_const[3];
+ ColumnPtr argument_columns[3];
+ for (int i = 0; i < argument_size; ++i) {
+ col_const[i] = is_column_const(*block.get_by_position(arguments[i]).column);
+ }
+ argument_columns[0] = col_const[0] ? static_cast<const ColumnConst&>(
+ *block.get_by_position(arguments[0]).column)
+ .convert_to_full_column()
+ : block.get_by_position(arguments[0]).column;
+
+ default_preprocess_parameter_columns(argument_columns, col_const, {1, 2}, block, arguments);
+
+ // the options have check in FE, so is always const, and get idx of 0
+ StringRef options_value;
+ if (argument_size == 4) {
+ options_value = block.get_by_position(arguments[3]).column->get_data_at(0);
+ }
+
+ if (col_const[1] && col_const[2]) {
+ Impl::execute_impl_const_args(context, argument_columns, options_value,
+ input_rows_count, result_data, result_offset,
+ result_null_map->get_data());
+ } else {
+ Impl::execute_impl(context, argument_columns, options_value, input_rows_count,
+ result_data, result_offset, result_null_map->get_data());
+ }
+
+ block.get_by_position(result).column =
+ ColumnNullable::create(std::move(result_data_column), std::move(result_null_map));
+ return Status::OK();
+ }
+
+ Status close(FunctionContext* context, FunctionContext::FunctionStateScope scope) override {
Review Comment:
recheck should we remove the func override
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2826419160
# BE UT Coverage Report
Increment line coverage `70.69% (82/116)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/03f436ed253e072e6f4ee8759493d0374dd8cffb_03f436ed253e072e6f4ee8759493d0374dd8cffb/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/03f436ed253e072e6f4ee8759493d0374dd8cffb_03f436ed253e072e6f4ee8759493d0374dd8cffb/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.60% (14555/27155) |
| Line Coverage | 42.45% (126359/297677) |
| Region Coverage | 41.24% (64543/156514) |
| Branch Coverage | 35.82% (32461/90610) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #50245:
URL: https://github.com/apache/doris/pull/50245#discussion_r2052204103
##########
be/src/vec/functions/function_regexp.cpp:
##########
@@ -52,12 +51,129 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct ThreeParamTypes {
+ static DataTypes get_variadic_argument_types() {
+ return {std::make_shared<DataTypeString>(), std::make_shared<DataTypeString>(),
+ std::make_shared<DataTypeString>()};
+ }
+};
+
+struct FourParamTypes {
+ static DataTypes get_variadic_argument_types() {
+ return {std::make_shared<DataTypeString>(), std::make_shared<DataTypeString>(),
+ std::make_shared<DataTypeString>(), std::make_shared<DataTypeString>()};
+ }
+};
+
+// template FunctionRegexpFunctionality is used for regexp_replace/regexp_replace_one
+template <typename Impl, typename ParamTypes>
+class FunctionRegexpReplace : public IFunction {
+public:
+ static constexpr auto name = Impl::name;
+
+ static FunctionPtr create() { return std::make_shared<FunctionRegexpReplace>(); }
+
+ String get_name() const override { return name; }
+
+ size_t get_number_of_arguments() const override {
+ return get_variadic_argument_types_impl().size();
+ }
+
+ bool is_variadic() const override { return true; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ return make_nullable(std::make_shared<DataTypeString>());
+ }
+
+ DataTypes get_variadic_argument_types_impl() const override {
+ return ParamTypes::get_variadic_argument_types();
+ }
+
+ Status open(FunctionContext* context, FunctionContext::FunctionStateScope scope) override {
+ if (scope == FunctionContext::THREAD_LOCAL) {
+ if (context->is_col_constant(1)) {
+ DCHECK(!context->get_function_state(scope));
+ const auto pattern_col = context->get_constant_col(1)->column_ptr;
+ const auto& pattern = pattern_col->get_data_at(0);
+ if (pattern.size == 0) {
+ return Status::OK();
+ }
+
+ std::string error_str;
+ std::unique_ptr<re2::RE2> scoped_re;
+ StringRef options_value;
+ if (context->get_num_args() == 4) {
+ DCHECK(context->is_col_constant(3));
+ const auto options_col = context->get_constant_col(3)->column_ptr;
+ options_value = options_col->get_data_at(0);
+ }
+
+ bool st = StringFunctions::compile_regex(pattern, &error_str, StringRef(),
+ options_value, scoped_re);
+ if (!st) {
+ context->set_error(error_str.c_str());
+ return Status::InvalidArgument(error_str);
+ }
+ std::shared_ptr<re2::RE2> re(scoped_re.release());
+ context->set_function_state(scope, re);
+ }
+ }
+ return Status::OK();
+ }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ uint32_t result, size_t input_rows_count) const override {
+ size_t argument_size = arguments.size();
+
+ auto result_null_map = ColumnUInt8::create(input_rows_count, 0);
+ auto result_data_column = ColumnString::create();
+ auto& result_data = result_data_column->get_chars();
+ auto& result_offset = result_data_column->get_offsets();
+ result_offset.resize(input_rows_count);
+
+ bool col_const[3];
+ ColumnPtr argument_columns[3];
+ for (int i = 0; i < argument_size; ++i) {
+ col_const[i] = is_column_const(*block.get_by_position(arguments[i]).column);
+ }
+ argument_columns[0] = col_const[0] ? static_cast<const ColumnConst&>(
+ *block.get_by_position(arguments[0]).column)
+ .convert_to_full_column()
+ : block.get_by_position(arguments[0]).column;
+
+ default_preprocess_parameter_columns(argument_columns, col_const, {1, 2}, block, arguments);
+
+ // the options have check in FE, so is always const, and get idx of 0
+ StringRef options_value;
+ if (argument_size == 4) {
Review Comment:
the call seems only in 157 line, `if (col_const[1] && col_const[2])` is true. no need call get_data_at
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2819930763
<details>
<summary>ClickBench: <b>Total hot run time: 29.22 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 855af6a806224444a79ca451a359844ceb1e325d, data reload: false
query1 0.04 0.04 0.03
query2 0.12 0.10 0.12
query3 0.25 0.19 0.20
query4 1.59 0.19 0.20
query5 0.60 0.57 0.57
query6 1.20 0.71 0.72
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.58 0.54 0.50
query10 0.57 0.56 0.56
query11 0.15 0.11 0.11
query12 0.15 0.11 0.12
query13 0.61 0.60 0.60
query14 1.19 1.17 1.20
query15 0.87 0.87 0.86
query16 0.37 0.39 0.36
query17 1.03 1.03 1.02
query18 0.21 0.20 0.20
query19 1.89 1.80 1.79
query20 0.01 0.02 0.01
query21 15.45 0.91 0.56
query22 0.74 1.21 0.71
query23 14.86 1.39 0.63
query24 6.92 2.35 0.38
query25 0.34 0.21 0.11
query26 0.64 0.16 0.14
query27 0.05 0.05 0.04
query28 9.25 0.85 0.45
query29 12.54 3.98 3.28
query30 0.26 0.09 0.06
query31 2.84 0.58 0.39
query32 3.23 0.55 0.46
query33 2.96 3.05 3.05
query34 15.82 5.14 4.48
query35 4.59 4.57 4.56
query36 0.65 0.49 0.48
query37 0.08 0.06 0.07
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.14 0.13
query41 0.08 0.02 0.02
query42 0.03 0.03 0.02
query43 0.04 0.03 0.03
Total cold run time: 103.11 s
Total hot run time: 29.22 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2819991873
<details>
<summary>TPC-H: <b>Total hot run time: 33983 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0532e1409c8a06da08431119687bc36b125c1c5b, data reload: false
------ Round 1 ----------------------------------
q1 26267 5137 5096 5096
q2 2073 293 187 187
q3 10376 1252 712 712
q4 10235 1021 545 545
q5 7541 2203 2416 2203
q6 183 162 133 133
q7 880 801 617 617
q8 9312 1309 1045 1045
q9 6981 5197 5121 5121
q10 6810 2329 1888 1888
q11 520 281 269 269
q12 348 363 221 221
q13 17763 3759 3133 3133
q14 229 219 212 212
q15 523 493 507 493
q16 452 449 412 412
q17 611 879 376 376
q18 7771 7117 7104 7104
q19 1227 934 578 578
q20 345 358 227 227
q21 4086 3349 2433 2433
q22 1078 980 978 978
Total cold run time: 115611 ms
Total hot run time: 33983 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5145 5132 5096 5096
q2 237 328 229 229
q3 2147 2667 2339 2339
q4 1470 1856 1465 1465
q5 4560 4425 4404 4404
q6 210 161 123 123
q7 1929 1925 1712 1712
q8 2600 2558 2474 2474
q9 7168 7117 7103 7103
q10 3028 3175 2747 2747
q11 555 511 487 487
q12 697 773 618 618
q13 3491 3937 3310 3310
q14 273 295 269 269
q15 532 491 489 489
q16 467 510 475 475
q17 1147 1593 1385 1385
q18 7562 7445 7494 7445
q19 846 813 935 813
q20 1956 2034 1913 1913
q21 5261 4650 4608 4608
q22 1057 1010 973 973
Total cold run time: 52338 ms
Total hot run time: 50477 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2822987120
<details>
<summary>ClickBench: <b>Total hot run time: 29.82 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 150af8994ad2ffe2aa230b879eca56b2b9a54c3c, data reload: false
query1 0.03 0.03 0.03
query2 0.13 0.10 0.10
query3 0.26 0.20 0.20
query4 1.59 0.19 0.20
query5 0.58 0.56 0.58
query6 1.20 0.71 0.73
query7 0.02 0.02 0.01
query8 0.04 0.03 0.04
query9 0.58 0.53 0.50
query10 0.58 0.57 0.58
query11 0.15 0.11 0.11
query12 0.14 0.11 0.11
query13 0.61 0.59 0.60
query14 1.15 1.18 1.17
query15 0.87 0.84 0.85
query16 0.39 0.38 0.38
query17 1.06 1.03 1.01
query18 0.22 0.20 0.20
query19 1.92 1.80 1.75
query20 0.01 0.01 0.01
query21 15.39 0.93 0.54
query22 0.76 1.07 0.65
query23 15.06 1.38 0.61
query24 7.12 1.19 1.58
query25 0.52 0.25 0.08
query26 0.54 0.16 0.14
query27 0.06 0.05 0.04
query28 9.27 0.78 0.43
query29 12.51 3.97 3.29
query30 0.25 0.09 0.07
query31 2.82 0.61 0.39
query32 3.22 0.53 0.46
query33 2.95 3.07 3.06
query34 15.77 5.11 4.51
query35 4.51 4.49 4.54
query36 0.64 0.51 0.48
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.03
query40 0.16 0.13 0.13
query41 0.08 0.02 0.02
query42 0.03 0.03 0.02
query43 0.04 0.04 0.03
Total cold run time: 103.4 s
Total hot run time: 29.82 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2819969319
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2826510554
# BE Regression P0 && UT Coverage Report
Increment line coverage `70.69% (82/116)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/50245_03f436ed253e072e6f4ee8759493d0374dd8cffb_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/50245_03f436ed253e072e6f4ee8759493d0374dd8cffb_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | |
| Line Coverage | |
| Region Coverage | |
| Branch Coverage | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2827211479
# BE Regression P0 && UT Coverage Report
Increment line coverage `70.69% (82/116)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/50245_03f436ed253e072e6f4ee8759493d0374dd8cffb_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/50245_03f436ed253e072e6f4ee8759493d0374dd8cffb_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | |
| Line Coverage | |
| Region Coverage | |
| Branch Coverage | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #50245:
URL: https://github.com/apache/doris/pull/50245#discussion_r2052204103
##########
be/src/vec/functions/function_regexp.cpp:
##########
@@ -52,12 +51,129 @@
namespace doris::vectorized {
#include "common/compile_check_begin.h"
+struct ThreeParamTypes {
+ static DataTypes get_variadic_argument_types() {
+ return {std::make_shared<DataTypeString>(), std::make_shared<DataTypeString>(),
+ std::make_shared<DataTypeString>()};
+ }
+};
+
+struct FourParamTypes {
+ static DataTypes get_variadic_argument_types() {
+ return {std::make_shared<DataTypeString>(), std::make_shared<DataTypeString>(),
+ std::make_shared<DataTypeString>(), std::make_shared<DataTypeString>()};
+ }
+};
+
+// template FunctionRegexpFunctionality is used for regexp_replace/regexp_replace_one
+template <typename Impl, typename ParamTypes>
+class FunctionRegexpReplace : public IFunction {
+public:
+ static constexpr auto name = Impl::name;
+
+ static FunctionPtr create() { return std::make_shared<FunctionRegexpReplace>(); }
+
+ String get_name() const override { return name; }
+
+ size_t get_number_of_arguments() const override {
+ return get_variadic_argument_types_impl().size();
+ }
+
+ bool is_variadic() const override { return true; }
+
+ DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
+ return make_nullable(std::make_shared<DataTypeString>());
+ }
+
+ DataTypes get_variadic_argument_types_impl() const override {
+ return ParamTypes::get_variadic_argument_types();
+ }
+
+ Status open(FunctionContext* context, FunctionContext::FunctionStateScope scope) override {
+ if (scope == FunctionContext::THREAD_LOCAL) {
+ if (context->is_col_constant(1)) {
+ DCHECK(!context->get_function_state(scope));
+ const auto pattern_col = context->get_constant_col(1)->column_ptr;
+ const auto& pattern = pattern_col->get_data_at(0);
+ if (pattern.size == 0) {
+ return Status::OK();
+ }
+
+ std::string error_str;
+ std::unique_ptr<re2::RE2> scoped_re;
+ StringRef options_value;
+ if (context->get_num_args() == 4) {
+ DCHECK(context->is_col_constant(3));
+ const auto options_col = context->get_constant_col(3)->column_ptr;
+ options_value = options_col->get_data_at(0);
+ }
+
+ bool st = StringFunctions::compile_regex(pattern, &error_str, StringRef(),
+ options_value, scoped_re);
+ if (!st) {
+ context->set_error(error_str.c_str());
+ return Status::InvalidArgument(error_str);
+ }
+ std::shared_ptr<re2::RE2> re(scoped_re.release());
+ context->set_function_state(scope, re);
+ }
+ }
+ return Status::OK();
+ }
+
+ Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+ uint32_t result, size_t input_rows_count) const override {
+ size_t argument_size = arguments.size();
+
+ auto result_null_map = ColumnUInt8::create(input_rows_count, 0);
+ auto result_data_column = ColumnString::create();
+ auto& result_data = result_data_column->get_chars();
+ auto& result_offset = result_data_column->get_offsets();
+ result_offset.resize(input_rows_count);
+
+ bool col_const[3];
+ ColumnPtr argument_columns[3];
+ for (int i = 0; i < argument_size; ++i) {
+ col_const[i] = is_column_const(*block.get_by_position(arguments[i]).column);
+ }
+ argument_columns[0] = col_const[0] ? static_cast<const ColumnConst&>(
+ *block.get_by_position(arguments[0]).column)
+ .convert_to_full_column()
+ : block.get_by_position(arguments[0]).column;
+
+ default_preprocess_parameter_columns(argument_columns, col_const, {1, 2}, block, arguments);
+
+ // the options have check in FE, so is always const, and get idx of 0
+ StringRef options_value;
+ if (argument_size == 4) {
Review Comment:
the call seems only in 157 line, `if (col_const[1] && col_const[2])` is true. no need call get_data_at, already do the work in open func
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2819907536
<details>
<summary>TPC-H: <b>Total hot run time: 34455 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 855af6a806224444a79ca451a359844ceb1e325d, data reload: false
------ Round 1 ----------------------------------
q1 26145 5121 5078 5078
q2 2056 272 184 184
q3 11297 1283 720 720
q4 10220 1043 534 534
q5 7541 2277 2405 2277
q6 190 165 131 131
q7 902 762 617 617
q8 10149 1297 1110 1110
q9 6966 5206 5170 5170
q10 6876 2318 1884 1884
q11 506 286 287 286
q12 379 361 223 223
q13 19582 4013 3414 3414
q14 251 232 228 228
q15 537 500 493 493
q16 458 468 416 416
q17 623 871 380 380
q18 8903 7384 7162 7162
q19 1962 999 559 559
q20 322 330 218 218
q21 3950 3403 2434 2434
q22 1071 1034 937 937
Total cold run time: 120886 ms
Total hot run time: 34455 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5186 5078 5086 5078
q2 228 329 229 229
q3 2108 2635 2419 2419
q4 1489 1928 1470 1470
q5 4487 4392 4355 4355
q6 212 168 128 128
q7 1958 1894 1774 1774
q8 2618 2561 2536 2536
q9 7204 7350 7248 7248
q10 3041 3263 2792 2792
q11 575 515 523 515
q12 708 772 693 693
q13 3627 3991 3292 3292
q14 280 302 274 274
q15 534 474 468 468
q16 477 499 462 462
q17 1157 1491 1413 1413
q18 7669 7583 7387 7387
q19 782 780 809 780
q20 1932 2001 1859 1859
q21 5358 4895 4774 4774
q22 1123 1058 1052 1052
Total cold run time: 52753 ms
Total hot run time: 50998 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2819861370
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2826343364
<details>
<summary>TPC-DS: <b>Total hot run time: 185893 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 03f436ed253e072e6f4ee8759493d0374dd8cffb, data reload: false
query1 1014 469 481 469
query2 6564 1820 1830 1820
query3 6739 219 215 215
query4 26511 23616 23541 23541
query5 4313 600 472 472
query6 308 211 222 211
query7 4615 479 268 268
query8 285 237 226 226
query9 8613 2559 2550 2550
query10 462 334 266 266
query11 15855 15001 14736 14736
query12 156 108 103 103
query13 1673 534 406 406
query14 9066 6089 6131 6089
query15 207 190 172 172
query16 7161 634 447 447
query17 1211 740 568 568
query18 1973 399 312 312
query19 191 189 161 161
query20 124 116 123 116
query21 212 127 115 115
query22 4347 4250 4268 4250
query23 33990 32883 33125 32883
query24 8489 2378 2415 2378
query25 534 471 393 393
query26 1245 276 149 149
query27 2747 500 326 326
query28 4315 2101 2096 2096
query29 783 542 427 427
query30 280 205 182 182
query31 942 866 751 751
query32 74 60 64 60
query33 552 357 301 301
query34 776 820 497 497
query35 787 791 726 726
query36 934 972 897 897
query37 113 101 76 76
query38 4189 4146 4049 4049
query39 1447 1398 1389 1389
query40 214 116 107 107
query41 63 58 57 57
query42 120 101 110 101
query43 477 498 462 462
query44 1276 785 794 785
query45 174 175 166 166
query46 845 1024 612 612
query47 1806 1837 1775 1775
query48 361 413 286 286
query49 783 494 431 431
query50 622 691 385 385
query51 4081 4153 4054 4054
query52 108 104 100 100
query53 232 251 183 183
query54 587 570 494 494
query55 89 79 80 79
query56 297 307 289 289
query57 1126 1141 1094 1094
query58 281 251 250 250
query59 2656 2654 2580 2580
query60 321 322 302 302
query61 129 133 146 133
query62 773 736 654 654
query63 224 186 183 183
query64 4429 1007 692 692
query65 4366 4302 4241 4241
query66 1152 397 304 304
query67 15659 15717 15370 15370
query68 8551 870 513 513
query69 475 299 257 257
query70 1202 1081 1133 1081
query71 465 319 288 288
query72 5736 4781 4877 4781
query73 742 661 349 349
query74 8877 9046 8681 8681
query75 3935 3181 2700 2700
query76 3643 1176 740 740
query77 787 373 287 287
query78 10009 10066 9294 9294
query79 2512 815 574 574
query80 635 533 435 435
query81 467 252 221 221
query82 446 126 100 100
query83 277 250 241 241
query84 353 101 82 82
query85 797 356 318 318
query86 346 319 306 306
query87 4441 4424 4392 4392
query88 2885 2225 2222 2222
query89 404 316 278 278
query90 1932 219 216 216
query91 153 136 114 114
query92 79 58 61 58
query93 1787 974 578 578
query94 665 420 308 308
query95 380 294 283 283
query96 480 558 274 274
query97 3179 3208 3160 3160
query98 224 210 215 210
query99 1451 1397 1290 1290
Total cold run time: 275363 ms
Total hot run time: 185893 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2818027740
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 commented on code in PR #50245:
URL: https://github.com/apache/doris/pull/50245#discussion_r2052371288
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/RegexpReplace.java:
##########
@@ -52,13 +59,44 @@ public RegexpReplace(Expression arg0, Expression arg1, Expression arg2) {
super("regexp_replace", arg0, arg1, arg2);
}
+ /**
+ * constructor with 4 arguments.
+ */
+ public RegexpReplace(Expression arg0, Expression arg1, Expression arg2, Expression arg3) {
+ super("regexp_replace", arg0, arg1, arg2, arg3);
+ }
+
/**
* withChildren.
*/
@Override
public RegexpReplace withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 3);
- return new RegexpReplace(children.get(0), children.get(1), children.get(2));
+ Preconditions.checkArgument(children.size() == 3 || children.size() == 4,
+ "RegexpReplace should have 3 or 4 children, but got: " + children.size());
+ if (children.size() == 3) {
+ return new RegexpReplace(children.get(0), children.get(1), children.get(2));
+ } else {
+ return new RegexpReplace(children.get(0), children.get(1), children.get(2), children.get(3));
+ }
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (children().size() == 3) {
+ return;
+ }
+ if (children().size() == 4) {
+ Expression value = child(3);
+ DataType type = value.getDataType();
+ if (!type.isStringLikeType()) {
+ throw new AnalysisException(
+ "The fourth param of regexp_replace must be a string type: " + this.toSql());
+ }
+ if (!value.isConstant()) {
Review Comment:
have changed, and I only want the fourth param is string literal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2819030226
<details>
<summary>TPC-H: <b>Total hot run time: 34221 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a52c323e58980ddf32fb800cc6885043380739b2, data reload: false
------ Round 1 ----------------------------------
q1 26191 5169 5045 5045
q2 2064 273 180 180
q3 10572 1257 691 691
q4 10238 1006 532 532
q5 7815 2429 2397 2397
q6 185 167 138 138
q7 923 756 641 641
q8 9304 1336 1151 1151
q9 6898 5103 5142 5103
q10 6854 2321 1892 1892
q11 498 278 282 278
q12 360 361 215 215
q13 17784 3736 3114 3114
q14 229 225 214 214
q15 523 484 482 482
q16 448 452 402 402
q17 606 875 366 366
q18 7561 7175 7142 7142
q19 2087 1046 544 544
q20 320 324 216 216
q21 4216 3480 2499 2499
q22 1030 992 979 979
Total cold run time: 116706 ms
Total hot run time: 34221 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5330 5138 5129 5129
q2 232 333 228 228
q3 2161 2651 2292 2292
q4 1426 1870 1458 1458
q5 4429 4480 4364 4364
q6 227 170 130 130
q7 1951 1949 1764 1764
q8 2619 2633 2604 2604
q9 7251 7064 7180 7064
q10 2987 3191 2731 2731
q11 592 521 495 495
q12 705 742 610 610
q13 3544 3865 3420 3420
q14 289 283 281 281
q15 512 492 476 476
q16 466 512 464 464
q17 1193 1511 1429 1429
q18 7672 7573 7429 7429
q19 816 884 1047 884
q20 1944 1956 1889 1889
q21 5287 4809 4844 4809
q22 1080 1076 1021 1021
Total cold run time: 52713 ms
Total hot run time: 50971 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 merged PR #50245:
URL: https://github.com/apache/doris/pull/50245
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2822950868
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2822970430
<details>
<summary>TPC-H: <b>Total hot run time: 33664 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 150af8994ad2ffe2aa230b879eca56b2b9a54c3c, data reload: false
------ Round 1 ----------------------------------
q1 26180 5014 5031 5014
q2 2062 273 183 183
q3 10403 1213 658 658
q4 10211 991 502 502
q5 7501 2385 2325 2325
q6 182 160 132 132
q7 930 736 612 612
q8 9314 1250 1029 1029
q9 6859 5060 5091 5060
q10 6955 2281 1879 1879
q11 483 272 278 272
q12 345 357 214 214
q13 17763 3671 3116 3116
q14 222 236 215 215
q15 532 477 489 477
q16 434 446 401 401
q17 591 847 354 354
q18 7726 7193 7084 7084
q19 1634 928 543 543
q20 326 329 226 226
q21 4085 2619 2411 2411
q22 1054 995 957 957
Total cold run time: 115792 ms
Total hot run time: 33664 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5060 5039 5022 5022
q2 233 318 228 228
q3 2161 2609 2275 2275
q4 1416 1782 1354 1354
q5 4482 4358 4414 4358
q6 212 167 123 123
q7 1975 1909 1730 1730
q8 2608 2533 2510 2510
q9 7270 7205 7049 7049
q10 2953 3225 2735 2735
q11 570 493 490 490
q12 672 768 622 622
q13 3484 3872 3318 3318
q14 287 334 278 278
q15 540 496 489 489
q16 455 492 465 465
q17 1159 1557 1353 1353
q18 7744 7563 7482 7482
q19 768 835 1009 835
q20 2015 1972 1828 1828
q21 5222 5014 4730 4730
q22 1102 1039 997 997
Total cold run time: 52388 ms
Total hot run time: 50271 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2819065280
<details>
<summary>ClickBench: <b>Total hot run time: 29.49 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a52c323e58980ddf32fb800cc6885043380739b2, data reload: false
query1 0.04 0.03 0.03
query2 0.12 0.11 0.11
query3 0.24 0.20 0.19
query4 1.60 0.20 0.19
query5 0.59 0.58 0.60
query6 1.19 0.73 0.72
query7 0.03 0.02 0.02
query8 0.04 0.04 0.04
query9 0.59 0.51 0.52
query10 0.57 0.57 0.57
query11 0.16 0.10 0.11
query12 0.14 0.11 0.12
query13 0.62 0.60 0.58
query14 1.15 1.21 1.23
query15 0.88 0.84 0.84
query16 0.39 0.39 0.38
query17 0.99 1.04 1.01
query18 0.21 0.19 0.19
query19 1.88 1.78 1.82
query20 0.01 0.01 0.01
query21 15.41 0.92 0.55
query22 0.75 1.34 0.78
query23 14.72 1.39 0.65
query24 7.71 1.10 0.69
query25 0.54 0.27 0.08
query26 0.49 0.15 0.13
query27 0.05 0.06 0.05
query28 10.15 0.88 0.44
query29 12.58 3.97 3.25
query30 0.27 0.09 0.06
query31 2.83 0.59 0.39
query32 3.22 0.55 0.45
query33 3.00 3.14 3.05
query34 15.66 5.08 4.48
query35 4.51 4.51 4.50
query36 0.66 0.51 0.47
query37 0.09 0.06 0.06
query38 0.06 0.04 0.04
query39 0.03 0.02 0.03
query40 0.18 0.15 0.12
query41 0.08 0.03 0.02
query42 0.04 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 104.51 s
Total hot run time: 29.49 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 commented on code in PR #50245:
URL: https://github.com/apache/doris/pull/50245#discussion_r2052370194
##########
regression-test/suites/nereids_syntax_p0/test_regexp_replace.groovy:
##########
@@ -36,4 +36,24 @@ suite("test_regexp_replace") {
(5, null);"""
qt_replace_in_table_chinese """SELECT id, regexp_replace(name, '\\\\p{Han}', '汉') as replaced_name FROM test_table_for_regexp;"""
+
+ qt_replace_ignore1 """select regexp_replace('{"abc":5},{"def":78}', '\\}\\,\\{', '\\}&&\\{', 'IGNORE_INVALID_ESCAPE');"""
+ qt_replace_ignore2 """select regexp_replace('abc', 'b', "\\}", 'IGNORE_INVALID_ESCAPE');"""
+ qt_replace_ignore3 """select regexp_replace_one('{"abc":5},{"def":78}', '\\}\\,\\{', '\\}&&\\{', 'IGNORE_INVALID_ESCAPE');"""
+ qt_replace_ignore4 """select regexp_replace_one('abc', 'b', '\\}', 'IGNORE_INVALID_ESCAPE');"""
Review Comment:
it's have register
`scalar(RegexpReplaceOne.class, "regexp_replace_one"),`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2818393051
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2819925203
<details>
<summary>TPC-DS: <b>Total hot run time: 193733 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 855af6a806224444a79ca451a359844ceb1e325d, data reload: false
query1 1407 1107 1064 1064
query2 6207 1824 1796 1796
query3 11097 4806 4734 4734
query4 25908 23769 23337 23337
query5 4262 642 476 476
query6 353 213 214 213
query7 3996 518 276 276
query8 307 259 238 238
query9 8491 2562 2603 2562
query10 479 324 287 287
query11 15565 15673 14975 14975
query12 168 113 111 111
query13 1710 534 411 411
query14 9959 6231 6283 6231
query15 202 192 171 171
query16 7588 691 508 508
query17 1284 744 610 610
query18 2107 432 315 315
query19 199 187 165 165
query20 133 129 132 129
query21 210 136 123 123
query22 4478 4592 4407 4407
query23 34585 33846 33586 33586
query24 8398 2424 2423 2423
query25 531 493 429 429
query26 802 287 165 165
query27 2821 544 347 347
query28 4636 2178 2168 2168
query29 662 630 475 475
query30 289 234 203 203
query31 879 903 777 777
query32 74 67 68 67
query33 571 373 315 315
query34 793 901 525 525
query35 816 860 798 798
query36 970 998 895 895
query37 116 105 83 83
query38 4370 4192 4203 4192
query39 1533 1494 1432 1432
query40 221 117 111 111
query41 54 52 50 50
query42 120 108 106 106
query43 488 532 505 505
query44 1292 822 824 822
query45 195 179 167 167
query46 841 1034 654 654
query47 1858 1843 1758 1758
query48 399 421 301 301
query49 732 539 437 437
query50 673 697 451 451
query51 4436 4237 4216 4216
query52 110 112 110 110
query53 233 257 187 187
query54 599 591 515 515
query55 84 82 80 80
query56 316 298 300 298
query57 1228 1209 1121 1121
query58 276 270 286 270
query59 2675 2803 2630 2630
query60 353 349 330 330
query61 133 128 148 128
query62 772 762 703 703
query63 226 193 189 189
query64 3229 1077 760 760
query65 4582 4411 4417 4411
query66 961 408 307 307
query67 16093 15505 15202 15202
query68 8706 909 514 514
query69 541 306 269 269
query70 1180 1141 1093 1093
query71 469 334 290 290
query72 5402 4782 4789 4782
query73 711 614 336 336
query74 8979 9104 8782 8782
query75 4074 3221 2729 2729
query76 3793 1228 773 773
query77 798 372 285 285
query78 10588 10607 9580 9580
query79 2267 818 573 573
query80 597 528 444 444
query81 486 270 222 222
query82 439 120 95 95
query83 287 255 230 230
query84 294 100 82 82
query85 809 361 390 361
query86 335 292 297 292
query87 4478 4464 4367 4367
query88 3231 2190 2170 2170
query89 389 317 273 273
query90 1932 205 214 205
query91 134 144 114 114
query92 76 59 56 56
query93 1644 973 583 583
query94 662 416 289 289
query95 378 287 279 279
query96 490 561 276 276
query97 3213 3275 3084 3084
query98 245 204 204 204
query99 1426 1417 1263 1263
Total cold run time: 281188 ms
Total hot run time: 193733 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2820002741
<details>
<summary>TPC-DS: <b>Total hot run time: 186171 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0532e1409c8a06da08431119687bc36b125c1c5b, data reload: false
query1 1023 475 533 475
query2 6548 1827 1782 1782
query3 6753 219 221 219
query4 26354 24204 23434 23434
query5 4404 637 479 479
query6 291 204 182 182
query7 4624 492 284 284
query8 283 236 235 235
query9 8613 2594 2614 2594
query10 473 329 264 264
query11 15404 15236 14809 14809
query12 152 113 107 107
query13 1660 535 405 405
query14 8971 6290 6326 6290
query15 207 194 173 173
query16 7133 612 446 446
query17 942 729 586 586
query18 1993 411 317 317
query19 199 189 176 176
query20 123 118 119 118
query21 219 127 107 107
query22 4003 4256 4129 4129
query23 34318 33101 33091 33091
query24 8503 2491 2392 2392
query25 535 442 387 387
query26 1239 268 150 150
query27 2757 490 332 332
query28 4313 2104 2089 2089
query29 777 551 426 426
query30 289 215 189 189
query31 906 893 767 767
query32 73 64 65 64
query33 566 379 315 315
query34 815 871 503 503
query35 783 806 741 741
query36 964 979 881 881
query37 117 105 77 77
query38 4150 4205 4164 4164
query39 1464 1399 1397 1397
query40 215 119 106 106
query41 57 52 52 52
query42 119 107 106 106
query43 507 511 474 474
query44 1330 795 793 793
query45 181 180 169 169
query46 861 1022 632 632
query47 1731 1786 1672 1672
query48 372 402 319 319
query49 773 522 410 410
query50 689 712 400 400
query51 4151 4125 4060 4060
query52 108 112 101 101
query53 236 261 198 198
query54 575 570 508 508
query55 89 81 80 80
query56 298 299 295 295
query57 1134 1143 1063 1063
query58 269 257 258 257
query59 2574 2705 2491 2491
query60 325 314 302 302
query61 147 125 125 125
query62 823 730 659 659
query63 240 187 199 187
query64 4385 990 662 662
query65 4365 4256 4255 4255
query66 1164 408 304 304
query67 15532 15418 15216 15216
query68 8215 891 515 515
query69 470 303 257 257
query70 1226 1176 1053 1053
query71 481 320 285 285
query72 5607 4688 4700 4688
query73 705 612 343 343
query74 8908 9132 8959 8959
query75 3932 3214 2697 2697
query76 3767 1202 779 779
query77 797 406 286 286
query78 9933 10227 9271 9271
query79 2001 827 567 567
query80 646 513 450 450
query81 468 253 223 223
query82 245 129 101 101
query83 287 253 236 236
query84 289 106 87 87
query85 744 410 320 320
query86 342 306 298 298
query87 4348 4486 4375 4375
query88 2811 2207 2239 2207
query89 390 309 282 282
query90 2080 209 216 209
query91 137 144 113 113
query92 73 63 59 59
query93 1369 973 582 582
query94 686 412 305 305
query95 380 295 280 280
query96 490 571 272 272
query97 3128 3198 3124 3124
query98 235 210 203 203
query99 1417 1434 1273 1273
Total cold run time: 272798 ms
Total hot run time: 186171 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2822982076
<details>
<summary>TPC-DS: <b>Total hot run time: 192199 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 150af8994ad2ffe2aa230b879eca56b2b9a54c3c, data reload: false
query1 1405 1071 1048 1048
query2 6062 1779 1754 1754
query3 11000 4756 4492 4492
query4 58108 25729 23573 23573
query5 5070 487 478 478
query6 383 210 197 197
query7 5067 491 291 291
query8 309 231 222 222
query9 6590 2535 2562 2535
query10 428 309 255 255
query11 15010 14946 14766 14766
query12 161 107 108 107
query13 1131 505 407 407
query14 10086 6369 6241 6241
query15 208 202 177 177
query16 7125 659 493 493
query17 1079 718 592 592
query18 1598 416 324 324
query19 200 202 180 180
query20 131 134 128 128
query21 203 120 149 120
query22 4406 4491 4288 4288
query23 34111 33578 33523 33523
query24 6523 2424 2433 2424
query25 466 481 396 396
query26 691 287 157 157
query27 2282 513 341 341
query28 3047 2145 2143 2143
query29 568 574 421 421
query30 277 221 190 190
query31 863 879 815 815
query32 69 66 61 61
query33 448 354 306 306
query34 773 873 521 521
query35 810 833 793 793
query36 986 1004 886 886
query37 118 98 75 75
query38 4295 4380 4169 4169
query39 1482 1436 1474 1436
query40 211 121 117 117
query41 54 51 52 51
query42 121 104 105 104
query43 487 497 471 471
query44 1358 821 804 804
query45 180 179 167 167
query46 859 1043 655 655
query47 1848 1870 1801 1801
query48 384 402 308 308
query49 702 523 449 449
query50 664 693 419 419
query51 4161 4198 4207 4198
query52 110 108 104 104
query53 243 259 184 184
query54 568 578 512 512
query55 84 84 86 84
query56 319 296 285 285
query57 1163 1170 1113 1113
query58 263 264 274 264
query59 2672 2748 2605 2605
query60 346 330 317 317
query61 160 146 149 146
query62 748 726 700 700
query63 222 191 192 191
query64 1610 1128 813 813
query65 4355 4208 4232 4208
query66 702 396 300 300
query67 15774 15628 15292 15292
query68 7900 879 517 517
query69 560 313 258 258
query70 1153 1114 1078 1078
query71 537 299 279 279
query72 5714 4735 4888 4735
query73 1311 669 343 343
query74 8939 9044 8650 8650
query75 3764 3173 2662 2662
query76 4311 1169 754 754
query77 598 366 294 294
query78 10076 10095 9290 9290
query79 3279 809 568 568
query80 653 504 430 430
query81 493 248 210 210
query82 497 122 91 91
query83 385 244 236 236
query84 297 94 79 79
query85 797 347 306 306
query86 419 293 285 285
query87 4350 4424 4301 4301
query88 3505 2203 2221 2203
query89 394 322 281 281
query90 1859 203 205 203
query91 143 151 111 111
query92 71 58 54 54
query93 2411 938 574 574
query94 657 406 292 292
query95 358 289 291 289
query96 486 560 268 268
query97 3174 3209 3121 3121
query98 221 202 207 202
query99 1465 1413 1248 1248
Total cold run time: 304714 ms
Total hot run time: 192199 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50245:
URL: https://github.com/apache/doris/pull/50245#issuecomment-2818027520
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](function) support regexp_replace function with ignore flag options [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #50245:
URL: https://github.com/apache/doris/pull/50245#discussion_r2052191078
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/RegexpReplace.java:
##########
@@ -52,13 +59,44 @@ public RegexpReplace(Expression arg0, Expression arg1, Expression arg2) {
super("regexp_replace", arg0, arg1, arg2);
}
+ /**
+ * constructor with 4 arguments.
+ */
+ public RegexpReplace(Expression arg0, Expression arg1, Expression arg2, Expression arg3) {
+ super("regexp_replace", arg0, arg1, arg2, arg3);
+ }
+
/**
* withChildren.
*/
@Override
public RegexpReplace withChildren(List<Expression> children) {
- Preconditions.checkArgument(children.size() == 3);
- return new RegexpReplace(children.get(0), children.get(1), children.get(2));
+ Preconditions.checkArgument(children.size() == 3 || children.size() == 4,
+ "RegexpReplace should have 3 or 4 children, but got: " + children.size());
+ if (children.size() == 3) {
+ return new RegexpReplace(children.get(0), children.get(1), children.get(2));
+ } else {
+ return new RegexpReplace(children.get(0), children.get(1), children.get(2), children.get(3));
+ }
+ }
+
+ @Override
+ public void checkLegalityBeforeTypeCoercion() {
+ if (children().size() == 3) {
+ return;
+ }
+ if (children().size() == 4) {
+ Expression value = child(3);
+ DataType type = value.getDataType();
+ if (!type.isStringLikeType()) {
+ throw new AnalysisException(
+ "The fourth param of regexp_replace must be a string type: " + this.toSql());
+ }
+ if (!value.isConstant()) {
Review Comment:
in planner, `isConstant` means there is not slot in the expr tree. so a function expr could return true when eval `isConstant`. if the last parameter could be a function, why not must be constant?
##########
regression-test/suites/nereids_syntax_p0/test_regexp_replace.groovy:
##########
@@ -36,4 +36,24 @@ suite("test_regexp_replace") {
(5, null);"""
qt_replace_in_table_chinese """SELECT id, regexp_replace(name, '\\\\p{Han}', '汉') as replaced_name FROM test_table_for_regexp;"""
+
+ qt_replace_ignore1 """select regexp_replace('{"abc":5},{"def":78}', '\\}\\,\\{', '\\}&&\\{', 'IGNORE_INVALID_ESCAPE');"""
+ qt_replace_ignore2 """select regexp_replace('abc', 'b', "\\}", 'IGNORE_INVALID_ESCAPE');"""
+ qt_replace_ignore3 """select regexp_replace_one('{"abc":5},{"def":78}', '\\}\\,\\{', '\\}&&\\{', 'IGNORE_INVALID_ESCAPE');"""
+ qt_replace_ignore4 """select regexp_replace_one('abc', 'b', '\\}', 'IGNORE_INVALID_ESCAPE');"""
Review Comment:
how this sql could exec since u not register regexp_replace_one in `fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org