You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "airborne12 (via GitHub)" <gi...@apache.org> on 2026/10/02 03:01:19 UTC
[PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
airborne12 opened a new pull request, #68701:
URL: https://github.com/apache/doris/pull/68701
### 问题
tokenize 的第二个参数通过 NereidsParser.parseProperties 解析。此前该入口仅匹配 propertyItemList，没有要求消费完整输入，导致合法属性后的非法 token 被静默忽略。例如：
```sql
SELECT tokenize('alpha beta', '"parser" = "unicode" trailing_tokens');
```
上述输入此前被接受，可能掩盖属性拼写或拼接错误。
### 改动
- 新增 propertyItemListWithEof 独立解析规则，要求属性列表后紧接 EOF，并让 parseProperties 使用该入口。
- 保留 SQL 内嵌 propertyItemList 的既有语法。合法单项、多项属性，以及末尾空白和注释仍可正常解析。
- 非法尾随标识符、分号、右括号、特殊符号或缺少逗号的属性被拒绝；tokenize 返回既有的属性格式错误。
- 补充 parser、Tokenize 单元测试及 test_tokenize 回归套件中的两个异常用例。
### 测试
- 修复前：`bash run-fe-ut.sh --run 'org.apache.doris.nereids.parser.NereidsParserTest#testParsePropertiesRejectsTrailingTokens'` 按预期失败：期望 ParseException，但没有抛出异常，确认复现。
- 修复后：`bash run-fe-ut.sh --run 'org.apache.doris.nereids.parser.NereidsParserTest,org.apache.doris.nereids.trees.expressions.functions.scalar.TokenizeTest'` 成功；共 96 项测试，失败、错误、跳过均为 0。
- `bash build.sh --fe` 成功，Checkstyle 零违规。
- `git diff --check` 通过；使用共享英文检查器的扫描逻辑检查相对 master 的完整工作区差异，无非英文描述。
- 新增回归用例尚未在集群运行，未手写或修改回归结果文件。
### 行为变化
含非法尾随 token 的 tokenize 属性字符串由静默接受变为明确报错。合法属性行为不变，无需文档变更。
### 提交范围
目标为 apache/doris:master。工作区仅有 5 个相关源码/测试文件的改动，未执行 git commit、push 或 gh pr create；请由平台提交、推送至 fork 并创建 PR。提交后发布前，请对实际提交执行共享 pr_english_check.py（--base master --head 提交SHA）；本轮遵照要求没有创建提交。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #68701:
URL: https://github.com/apache/doris/pull/68701#issuecomment-6051620841
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68701:
URL: https://github.com/apache/doris/pull/68701#discussion_r4215623180
##########
fe/fe-core/src/test/java/org/apache/doris/nereids/parser/NereidsParserTest.java:
##########
@@ -99,6 +99,26 @@
public class NereidsParserTest extends ParserTestBase {
+ @Test
+ public void testParsePropertiesRejectsTrailingTokens() {
+ NereidsParser parser = new NereidsParser();
+ for (String suffix : new String[] {" trailing_tokens", " ;", " )", " @", " \"extra\"=\"value\""}) {
+ Assertions.assertThrows(ParseException.class,
+ () -> parser.parseProperties("\"parser\" = \"unicode\"" + suffix), suffix);
+ }
+ }
+
+ @Test
+ public void testParseProperties() {
+ NereidsParser parser = new NereidsParser();
+ Assertions.assertEquals(ImmutableMap.of("parser", "unicode"),
+ parser.parseProperties("\"parser\" = \"unicode\""));
+ Assertions.assertEquals(ImmutableMap.of("parser", "unicode", "support_phrase", "true"),
+ parser.parseProperties("\"parser\" = \"unicode\", \"support_phrase\" = \"true\""));
+ Assertions.assertEquals(ImmutableMap.of("parser", "unicode"),
+ parser.parseProperties(" \"parser\" = \"unicode\" /* comment */ \n -- comment\n"));
Review Comment:
[P2] Keep comment assignments out of BE property parsing. This new test accepts trailing comments, but FE ignores them while BE regex-scans the original `tokenize` string. With visible `"parser"="unicode","char_filter_type"="char_replace","char_filter_pattern"=".","char_filter_replacement"="_"` followed by `/* "char_filter_replacement"="xyz" */`, FE validates `_`, then BE overwrites it with `xyz` and applies `x` as the replacement. A commented property thus changes results and bypasses FE validation. Please make FE validation and BE parsing agree on comments and cover an assignment inside a comment in an execution test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68701:
URL: https://github.com/apache/doris/pull/68701#issuecomment-5944824955
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68701:
URL: https://github.com/apache/doris/pull/68701#issuecomment-6056613429
# FE Regression Coverage Report
Increment line coverage ` 1.15% (1/87)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68701_d6a57457f24f25591184047fe4b0b94ae87b232b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68701_d6a57457f24f25591184047fe4b0b94ae87b232b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #68701:
URL: https://github.com/apache/doris/pull/68701#issuecomment-6051624505
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #68701:
URL: https://github.com/apache/doris/pull/68701#issuecomment-6053649654
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on PR #68701:
URL: https://github.com/apache/doris/pull/68701#issuecomment-6053652847
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 merged PR #68701:
URL: https://github.com/apache/doris/pull/68701
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68701:
URL: https://github.com/apache/doris/pull/68701#issuecomment-6054263628
<details>
<summary>TPC-DS: <b>Total hot run time: 152243 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d6a57457f24f25591184047fe4b0b94ae87b232b, data reload: false
query5 4295 576 477 477
query6 427 207 188 188
query7 4795 442 224 224
query8 328 196 175 175
query9 8789 4022 4002 4002
query10 493 342 267 267
query11 5980 3441 3182 3182
query12 148 89 85 85
query13 1267 437 325 325
query14 6566 4737 4453 4453
query14_1 4295 4280 4244 4244
query15 208 221 189 189
query16 948 432 422 422
query17 862 668 554 554
query18 2430 417 311 311
query19 198 177 140 140
query20 80 80 77 77
query21 212 142 119 119
query22 13010 12949 12920 12920
query23 13171 12451 12057 12057
query23_1 12189 12291 12210 12210
query24 7088 820 457 457
query24_1 478 464 473 464
query25 550 421 359 359
query26 1254 258 141 141
query27 2773 429 269 269
query28 4602 1913 1902 1902
query29 1582 592 445 445
query30 300 226 188 188
query31 907 777 656 656
query32 149 95 112 95
query33 515 310 253 253
query34 892 851 498 498
query35 762 756 646 646
query36 894 800 719 719
query37 137 103 89 89
query38 1819 1746 1722 1722
query39 741 675 686 675
query39_1 655 690 684 684
query40 231 128 103 103
query41 72 69 67 67
query42 85 82 86 82
query43 368 385 331 331
query44 1323 696 702 696
query45 187 188 166 166
query46 882 975 568 568
query47 2695 3014 2949 2949
query48 292 299 206 206
query49 569 389 309 309
query50 692 275 210 210
query51 10162 10526 10378 10378
query52 87 79 69 69
query53 187 210 152 152
query54 231 188 180 180
query55 72 77 64 64
query56 240 230 194 194
query57 1622 1585 1562 1562
query58 274 258 241 241
query59 2375 2345 2131 2131
query60 309 239 216 216
query61 140 146 147 146
query62 401 335 280 280
query63 192 158 172 158
query64 2742 941 780 780
query65 3421 3362 3391 3362
query66 1781 412 299 299
query67 20210 20225 20241 20225
query68 2917 1021 634 634
query69 462 290 245 245
query70 948 866 874 866
query71 307 230 208 208
query72 2657 2567 2206 2206
query73 528 525 295 295
query74 4614 4478 4316 4316
query75 2343 2292 1906 1906
query76 2925 1017 587 587
query77 355 387 299 299
query78 9010 8936 8503 8503
query79 970 903 506 506
query80 783 427 374 374
query81 541 326 275 275
query82 590 143 114 114
query83 288 196 180 180
query84 315 118 97 97
query85 938 430 393 393
query86 359 240 232 232
query87 2049 1933 1825 1825
query88 3646 2691 2683 2683
query89 344 292 263 263
query90 1718 185 186 185
query91 161 150 126 126
query92 101 86 81 81
query93 1016 1008 559 559
query94 491 293 271 271
query95 597 352 378 352
query96 658 512 232 232
query97 2442 2416 2311 2311
query98 158 148 144 144
query99 784 769 640 640
Total cold run time: 234376 ms
Total hot run time: 152243 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68701:
URL: https://github.com/apache/doris/pull/68701#issuecomment-6054352133
<details>
<summary>ClickBench: <b>Total hot run time: 24.54 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d6a57457f24f25591184047fe4b0b94ae87b232b, data reload: false
query1 0.00 0.00 0.00
query2 0.10 0.05 0.05
query3 0.27 0.13 0.12
query4 1.62 0.13 0.13
query5 0.28 0.26 0.25
query6 1.16 0.70 0.67
query7 0.04 0.01 0.01
query8 0.06 0.04 0.04
query9 0.42 0.37 0.35
query10 0.59 0.59 0.57
query11 0.22 0.14 0.14
query12 0.19 0.14 0.14
query13 0.49 0.50 0.48
query14 0.97 0.97 0.97
query15 0.63 0.60 0.60
query16 0.37 0.36 0.35
query17 1.17 1.15 1.15
query18 0.23 0.21 0.20
query19 2.10 1.94 1.97
query20 0.02 0.02 0.01
query21 15.50 0.28 0.14
query22 4.46 0.05 0.05
query23 16.28 0.31 0.12
query24 3.10 0.59 0.46
query25 0.11 0.05 0.05
query26 1.24 0.24 0.15
query27 0.06 0.04 0.03
query28 2.43 0.60 0.42
query29 12.54 4.16 3.25
query30 0.28 0.14 0.13
query31 2.78 0.58 0.34
query32 3.22 0.62 0.52
query33 3.23 3.26 3.20
query34 15.43 4.00 3.37
query35 3.33 3.34 3.38
query36 0.56 0.44 0.48
query37 0.10 0.07 0.06
query38 0.05 0.04 0.04
query39 0.03 0.04 0.03
query40 0.18 0.15 0.15
query41 0.09 0.03 0.03
query42 0.04 0.02 0.03
query43 0.04 0.04 0.03
Total cold run time: 96.01 s
Total hot run time: 24.54 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68701:
URL: https://github.com/apache/doris/pull/68701#issuecomment-6057009002
# FE UT Coverage Report
Increment line coverage ` 100.00% (1/1)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68701_d6a57457f24f25591184047fe4b0b94ae87b232b/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68701_d6a57457f24f25591184047fe4b0b94ae87b232b/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](fe) Reject trailing tokens in tokenize properties [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68701:
URL: https://github.com/apache/doris/pull/68701#issuecomment-6054109496
<details>
<summary>TPC-H: <b>Total hot run time: 28902 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d6a57457f24f25591184047fe4b0b94ae87b232b, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17623 5706 5519 5519
q2 2163 342 238 238
q3 10285 1390 832 832
q4 4700 464 351 351
q5 7429 779 530 530
q6 182 187 155 155
q7 754 794 602 602
q8 9256 1298 1056 1056
q9 5674 4459 4436 4436
q10 6799 1286 1018 1018
q11 436 258 229 229
q12 629 406 289 289
q13 18062 3072 2381 2381
q14 281 269 243 243
q15 q16 733 722 675 675
q17 1176 735 566 566
q18 7197 6259 6170 6170
q19 1148 940 581 581
q20 386 334 239 239
q21 5064 2513 2565 2513
q22 389 357 279 279
Total cold run time: 100366 ms
Total hot run time: 28902 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6509 6454 6414 6414
q2 791 608 582 582
q3 5587 5751 5373 5373
q4 2068 2142 1557 1557
q5 5905 5635 5642 5635
q6 249 186 147 147
q7 2302 2118 1844 1844
q8 3076 2725 2784 2725
q9 8177 8036 7732 7732
q10 3669 3584 3217 3217
q11 592 401 387 387
q12 663 695 501 501
q13 2757 3075 2372 2372
q14 284 283 258 258
q15 q16 674 683 627 627
q17 7737 6978 6802 6802
q18 13166 12273 13065 12273
q19 912 848 774 774
q20 2200 2166 1940 1940
q21 5864 4812 4791 4791
q22 504 482 393 393
Total cold run time: 73686 ms
Total hot run time: 66344 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org