You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "JamesIsHuang (via GitHub)" <gi...@apache.org> on 2026/03/05 06:45:55 UTC
[PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
JamesIsHuang opened a new pull request, #61061:
URL: https://github.com/apache/doris/pull/61061
## Proposed changes
Issue: https://github.com/apache/doris/issues/60879
Implement `ANSI_QUOTES` SQL mode support in the Nereids parser. When enabled via `SET sql_mode = 'ANSI_QUOTES'`, double-quoted strings (`"identifier"`) are treated as identifiers instead of string literals, matching standard SQL and MySQL behavior.
### Changes
- **DorisLexer.g4**: Add `isAnsiQuotes` flag and `DOUBLE_QUOTED_IDENTIFIER` lexer token with semantic predicate; gate the double-quote alternative in `STRING_LITERAL` with `!isAnsiQuotes`
- **DorisParser.g4**: Add `DOUBLE_QUOTED_IDENTIFIER` as an alternative in the `quotedIdentifier` parser rule
- **NereidsParser.java**: Pass `isAnsiQuotes` flag from `SqlModeHelper` to the lexer in both `scan()` and `parseAllTokens()`
- **PostProcessor.java**: Handle `DOUBLE_QUOTED_IDENTIFIER` token in `exitQuotedIdentifier()` — strip surrounding double quotes and unescape `""` to `"`
- **SqlModeHelper.java**: Add `hasAnsiQuotes()` helper method
- **NereidsParserTest.java**: Add unit tests for ANSI_QUOTES on/off behavior
### How it works
1. User sets `SET sql_mode = 'ANSI_QUOTES';`
2. `SqlModeHelper.hasAnsiQuotes()` returns `true`
3. `NereidsParser` sets `lexer.isAnsiQuotes = true`
4. Lexer matches `"col_name"` as `DOUBLE_QUOTED_IDENTIFIER` instead of `STRING_LITERAL`
5. Parser treats it as a `quotedIdentifier` (same as backtick-quoted)
6. `PostProcessor` strips quotes and unescapes `""` -> `"`
### Behavior
| SQL Mode | `"hello"` parsed as | `'hello'` parsed as | `` `hello` `` parsed as |
|---|---|---|---|
| Default | String literal | String literal | Identifier |
| ANSI_QUOTES | **Identifier** | String literal | Identifier |
## Checklist
- [x] Does it affect the results of queries? No (only changes how double quotes are interpreted based on sql_mode)
- [x] Have you added test cases? Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-4005164292
<details>
<summary>TPC-H: <b>Total hot run time: 28144 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 954d11fb2fc4b736e144cfd47f97d0b98c818412, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17674 4565 4302 4302
q2 q3 10653 802 523 523
q4 4688 376 264 264
q5 7577 1215 1040 1040
q6 197 189 162 162
q7 813 895 719 719
q8 9350 1516 1379 1379
q9 4917 4782 4749 4749
q10 6346 1923 1710 1710
q11 495 288 263 263
q12 772 593 487 487
q13 18054 2943 2188 2188
q14 244 257 233 233
q15 1024 867 856 856
q16 782 763 704 704
q17 733 892 411 411
q18 5785 5425 5261 5261
q19 1159 1043 658 658
q20 519 521 422 422
q21 4531 3595 1472 1472
q22 418 490 341 341
Total cold run time: 96731 ms
Total hot run time: 28144 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4662 4561 4553 4553
q2 q3 4027 4421 4062 4062
q4 894 1229 834 834
q5 4075 4475 4310 4310
q6 194 307 152 152
q7 1943 1693 1530 1530
q8 2615 2735 2631 2631
q9 7532 7351 7440 7351
q10 3747 4093 3636 3636
q11 698 671 639 639
q12 699 668 555 555
q13 2731 3215 2358 2358
q14 344 389 458 389
q15 1067 1023 928 928
q16 789 820 802 802
q17 1223 1463 1479 1463
q18 8084 6796 6777 6777
q19 1126 907 1081 907
q20 2135 2153 2020 2020
q21 4289 3558 3376 3376
q22 503 493 708 493
Total cold run time: 53377 ms
Total hot run time: 49766 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "JamesIsHuang (via GitHub)" <gi...@apache.org>.
JamesIsHuang commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-4015326465
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-4016772376
# FE UT Coverage Report
Increment line coverage ` 84.62% (22/26)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61061_79e48db38f9dee63702e51e8d4bcd20368dce5f6/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61061_79e48db38f9dee63702e51e8d4bcd20368dce5f6/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-4010190558
<details>
<summary>TPC-DS: <b>Total hot run time: 158386 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66084c3d87c7430784f8a27526de5a3fb24c073e, data reload: false
query5 4542 811 708 708
query6 373 274 249 249
query7 4248 515 296 296
query8 511 431 412 412
query9 8787 2808 2827 2808
query10 580 447 396 396
query11 7406 5901 5638 5638
query12 217 158 163 158
query13 1322 513 409 409
query14 6060 4013 3724 3724
query14_1 2993 2882 2891 2882
query15 213 209 195 195
query16 1035 501 465 465
query17 1129 739 627 627
query18 2674 484 391 391
query19 229 227 200 200
query20 163 154 148 148
query21 265 177 162 162
query22 4909 5067 5031 5031
query23 16037 15637 15403 15403
query23_1 16103 16083 15828 15828
query24 7488 1983 1300 1300
query24_1 1313 1332 1323 1323
query25 707 871 484 484
query26 1355 366 194 194
query27 3962 501 322 322
query28 4859 2651 2188 2188
query29 1184 731 587 587
query30 450 292 250 250
query31 1473 1410 1362 1362
query32 108 101 307 101
query33 597 393 445 393
query34 1579 1041 668 668
query35 709 744 660 660
query36 1193 1599 1097 1097
query37 213 185 103 103
query38 3080 3200 2964 2964
query39 951 1172 893 893
query39_1 898 860 859 859
query40 259 182 169 169
query41 139 139 163 139
query42 326 317 323 317
query43 279 287 256 256
query44
query45 220 212 209 209
query46 904 1023 638 638
query47 2206 2255 2111 2111
query48 351 379 271 271
query49 769 626 521 521
query50 729 332 257 257
query51 4209 4228 4096 4096
query52 305 311 297 297
query53 331 374 318 318
query54 346 323 317 317
query55 108 102 97 97
query56 366 379 360 360
query57 1419 1415 1344 1344
query58 362 352 346 346
query59 1366 1530 1335 1335
query60 392 397 365 365
query61 185 177 180 177
query62 677 631 580 580
query63 350 319 320 319
query64 5108 1387 1085 1085
query65
query66 1712 773 648 648
query67 16471 16349 16242 16242
query68
query69 426 349 329 329
query70 936 1038 987 987
query71 378 342 344 342
query72 2882 2692 2468 2468
query73 573 614 369 369
query74 10017 9992 9802 9802
query75 2918 2814 2521 2521
query76 2318 1067 711 711
query77 472 535 419 419
query78 11323 11333 10762 10762
query79 3193 820 636 636
query80 1859 705 616 616
query81 624 307 280 280
query82 1027 172 135 135
query83 384 326 294 294
query84 281 134 114 114
query85 960 530 507 507
query86 509 334 313 313
query87 3232 3133 3014 3014
query88 3735 2841 2839 2839
query89 463 407 372 372
query90 2206 211 211 211
query91 196 184 162 162
query92 101 95 94 94
query93 1799 843 527 527
query94 663 352 303 303
query95 629 371 343 343
query96 643 525 244 244
query97 2551 2515 2473 2473
query98 259 248 247 247
query99 1051 1045 967 967
Total cold run time: 245092 ms
Total hot run time: 158386 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-4005231074
<details>
<summary>TPC-DS: <b>Total hot run time: 158131 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 954d11fb2fc4b736e144cfd47f97d0b98c818412, data reload: false
query5 4492 818 684 684
query6 377 273 251 251
query7 4244 497 288 288
query8 500 419 406 406
query9 8757 2809 2805 2805
query10 556 444 411 411
query11 7494 5914 5720 5720
query12 217 156 151 151
query13 1325 540 411 411
query14 5993 4027 3646 3646
query14_1 2898 2877 2856 2856
query15 218 213 193 193
query16 1024 491 458 458
query17 1131 739 635 635
query18 2662 479 391 391
query19 223 229 201 201
query20 160 153 152 152
query21 261 176 165 165
query22 4840 4873 5090 4873
query23 18640 16093 15622 15622
query23_1 16097 15797 15907 15797
query24 7963 1700 1421 1421
query24_1 1284 1370 1356 1356
query25 594 877 623 623
query26 1381 303 184 184
query27 3255 587 310 310
query28 4504 1936 1963 1936
query29 873 587 512 512
query30 336 269 243 243
query31 1422 1369 1297 1297
query32 105 92 88 88
query33 558 382 346 346
query34 954 935 589 589
query35 673 724 645 645
query36 1112 1160 1039 1039
query37 149 114 104 104
query38 2929 3019 2913 2913
query39 947 907 886 886
query39_1 871 860 870 860
query40 268 189 168 168
query41 144 140 138 138
query42 317 321 315 315
query43 280 289 278 278
query44
query45 219 213 212 212
query46 900 1000 640 640
query47 2194 2236 2109 2109
query48 364 386 269 269
query49 780 644 518 518
query50 728 337 264 264
query51 4141 4151 4104 4104
query52 305 314 302 302
query53 324 370 323 323
query54 339 314 321 314
query55 106 109 102 102
query56 372 372 365 365
query57 1452 1436 1328 1328
query58 354 346 345 345
query59 1367 1511 1342 1342
query60 381 379 363 363
query61 196 194 190 190
query62 675 619 582 582
query63 349 324 313 313
query64 5204 1367 1142 1142
query65
query66 1702 750 643 643
query67 16373 16387 16381 16381
query68
query69 427 332 334 332
query70 1026 1030 912 912
query71 369 344 327 327
query72 2862 2786 2428 2428
query73 583 583 359 359
query74 10027 9970 9836 9836
query75 2931 2830 2555 2555
query76 2358 1056 696 696
query77 455 490 414 414
query78 11342 11403 10790 10790
query79 1189 815 642 642
query80 1426 720 649 649
query81 581 308 282 282
query82 1030 170 134 134
query83 378 325 293 293
query84 272 142 116 116
query85 965 548 504 504
query86 451 360 326 326
query87 3147 3165 3027 3027
query88 3714 2806 2814 2806
query89 454 403 390 390
query90 2058 208 204 204
query91 189 186 160 160
query92 99 97 89 89
query93 1016 857 526 526
query94 662 368 284 284
query95 613 361 350 350
query96 647 528 247 247
query97 2507 2562 2454 2454
query98 262 247 244 244
query99 1056 1047 974 974
Total cold run time: 241871 ms
Total hot run time: 158131 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "JamesIsHuang (via GitHub)" <gi...@apache.org>.
JamesIsHuang commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-4015338354
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "JamesIsHuang (via GitHub)" <gi...@apache.org>.
JamesIsHuang commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-4002887919
Hi @morrySnow, could you please help review this PR? This implements `ANSI_QUOTES` SQL mode support (issue #60879), following a similar pattern to the `NO_BACKSLASH_ESCAPES` implementation you did in #53449. The changes are in the Nereids lexer/parser layer. Thanks!
cc @lucienlgl (issue author)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-4002666039
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "JamesIsHuang (via GitHub)" <gi...@apache.org>.
JamesIsHuang commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-4010031140
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "JamesIsHuang (via GitHub)" <gi...@apache.org>.
JamesIsHuang commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-4004953682
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-4010152700
<details>
<summary>TPC-H: <b>Total hot run time: 28223 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66084c3d87c7430784f8a27526de5a3fb24c073e, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17663 4451 4309 4309
q2 q3 10651 805 524 524
q4 4694 377 260 260
q5 7558 1227 1029 1029
q6 192 191 167 167
q7 833 886 719 719
q8 9381 1510 1344 1344
q9 4939 4948 4676 4676
q10 6345 1952 1690 1690
q11 509 293 279 279
q12 776 601 513 513
q13 18066 2930 2236 2236
q14 255 257 249 249
q15 998 871 878 871
q16 799 754 691 691
q17 723 885 424 424
q18 5942 5333 5316 5316
q19 1218 1043 663 663
q20 528 536 421 421
q21 4706 3521 1499 1499
q22 396 421 343 343
Total cold run time: 97172 ms
Total hot run time: 28223 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4748 4535 4554 4535
q2 q3 3912 4445 3804 3804
q4 918 1263 913 913
q5 4378 4409 4538 4409
q6 363 192 317 192
q7 1832 1672 1713 1672
q8 2515 2739 2561 2561
q9 7827 7346 7363 7346
q10 3778 4007 3671 3671
q11 563 473 455 455
q12 538 629 485 485
q13 2792 3121 2423 2423
q14 370 311 284 284
q15 918 848 956 848
q16 805 806 788 788
q17 1160 1422 1345 1345
q18 7571 6889 6753 6753
q19 1234 969 997 969
q20 2321 2199 2073 2073
q21 4060 3674 3465 3465
q22 559 463 412 412
Total cold run time: 53162 ms
Total hot run time: 49403 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #61061: [Feature](parser) Support SQL Mode: ANSI_QUOTES
URL: https://github.com/apache/doris/pull/61061
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Feature](parser) Support SQL Mode: ANSI_QUOTES [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61061:
URL: https://github.com/apache/doris/pull/61061#issuecomment-5533842576
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org