You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2026/05/22 04:04:52 UTC
[PR] [fix](be) Escape JSONB path member control characters [doris]
mrhhsg opened a new pull request, #63517:
URL: https://github.com/apache/doris/pull/63517
### What problem does this PR solve?
Issue Number: None
Problem Summary: JSONB path serialization emitted raw control characters in member names. This could make paths returned by JSON functions or shown in diagnostics contain raw newlines/tabs and fail path round-trip parsing. Escape JSONB path member names with JSON-compatible sequences and decode those sequences while parsing path members.
### Release note
None
### Check List (For Author)
- Test:
- Unit Test: `./run-be-ut.sh --run --filter=JsonbDocumentTest.jsonb_path_member_to_string_escapes_control_characters -j 16`
- Format check: `build-support/clang-format.sh`; `build-support/check-format.sh`; `git diff --check`
- Static Analysis: attempted `build-support/run-clang-tidy.sh --base d7d516ff7b60e2ff6971e7d18cd805478595b5c2 --build-dir be/ut_build_ASAN`, blocked by existing clang-tidy/toolchain diagnostics outside the change
- Behavior changed: Yes. JSONB path member serialization now escapes control characters instead of emitting raw control bytes.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4581611565
<details>
<summary>TPC-H: <b>Total hot run time: 31882 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2cb8f4c0b242cf643c5cfca3f7af047058786467, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17606 4091 4016 4016
q2 q3 10742 1471 834 834
q4 4706 485 350 350
q5 8057 2310 2177 2177
q6 386 177 138 138
q7 1030 785 642 642
q8 9618 1788 1738 1738
q9 7206 4998 5012 4998
q10 6459 2260 1925 1925
q11 437 285 248 248
q12 684 438 310 310
q13 18219 3407 2835 2835
q14 269 260 238 238
q15 q16 816 786 713 713
q17 995 891 1002 891
q18 6922 5750 5680 5680
q19 1155 1344 1140 1140
q20 510 408 259 259
q21 5660 2687 2442 2442
q22 437 361 308 308
Total cold run time: 101914 ms
Total hot run time: 31882 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4565 4412 4302 4302
q2 q3 4562 4934 4400 4400
q4 2179 2330 1370 1370
q5 4528 4341 5245 4341
q6 250 195 151 151
q7 2004 1858 1625 1625
q8 2580 2276 2188 2188
q9 8123 8105 8087 8087
q10 4903 4766 4336 4336
q11 595 417 429 417
q12 768 747 552 552
q13 3344 3725 3043 3043
q14 294 319 291 291
q15 q16 707 730 681 681
q17 1376 1371 1352 1352
q18 8089 7372 6884 6884
q19 1159 1107 1128 1107
q20 2222 2231 1935 1935
q21 5440 4953 4627 4627
q22 543 470 412 412
Total cold run time: 58231 ms
Total hot run time: 52101 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63517:
URL: https://github.com/apache/doris/pull/63517#discussion_r3323838482
##########
be/src/util/jsonb_document.h:
##########
@@ -363,19 +418,51 @@ struct leg_info {
Review Comment:
Thanks, fixed in `2cb8f4c0b24`. `has_escapes` is now reset at the start of each parsed path leg, so a following unescaped member no longer calls `remove_escapes()` and overwrites the next delimiter. I also added `jsonb_path_escaped_member_round_trips_with_following_legs` to cover `$.
"a\\nb".c.d`-style nested round trip with trailing legs.
Validation:
- `build-support/clang-format.sh`; `build-support/check-format.sh`; `git diff --check`
- `DORIS_HOME=$PWD ninja -C be/ut_build_ASAN test/CMakeFiles/doris_be_test.dir/core/jsonb/jsonb_document_test.cpp.o`
- standalone JsonbPath nested escaped-member repro compiled and ran successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4550564611
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4581635922
<details>
<summary>TPC-DS: <b>Total hot run time: 172100 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2cb8f4c0b242cf643c5cfca3f7af047058786467, data reload: false
query5 4328 661 541 541
query6 328 224 204 204
query7 4251 551 312 312
query8 322 227 220 220
query9 8795 4157 4057 4057
query10 452 343 306 306
query11 5766 2436 2170 2170
query12 184 131 129 129
query13 1312 621 429 429
query14 6109 5514 5181 5181
query14_1 4521 4503 4527 4503
query15 217 216 194 194
query16 1064 482 468 468
query17 1153 756 617 617
query18 2790 492 375 375
query19 235 212 179 179
query20 141 134 131 131
query21 218 157 128 128
query22 13661 13538 13381 13381
query23 17640 16549 16147 16147
query23_1 16334 16406 16283 16283
query24 7509 1824 1291 1291
query24_1 1325 1328 1341 1328
query25 565 483 412 412
query26 1173 329 178 178
query27 2685 575 332 332
query28 4383 2049 2016 2016
query29 1024 625 510 510
query30 314 235 191 191
query31 1151 1077 968 968
query32 92 76 72 72
query33 537 358 302 302
query34 1198 1142 657 657
query35 767 799 715 715
query36 1441 1440 1199 1199
query37 157 107 91 91
query38 3197 3193 3089 3089
query39 946 912 896 896
query39_1 879 892 901 892
query40 226 159 129 129
query41 68 64 62 62
query42 115 110 111 110
query43 339 332 298 298
query44
query45 216 204 207 204
query46 1084 1184 745 745
query47 2358 2417 2258 2258
query48 414 421 307 307
query49 636 499 398 398
query50 1021 347 265 265
query51 4300 4298 4319 4298
query52 107 107 95 95
query53 259 284 203 203
query54 339 281 271 271
query55 96 91 89 89
query56 313 296 315 296
query57 1425 1415 1350 1350
query58 313 280 277 277
query59 1609 1677 1501 1501
query60 331 323 320 320
query61 168 164 163 163
query62 696 658 596 596
query63 252 202 213 202
query64 2207 817 645 645
query65
query66 1615 504 395 395
query67 29745 29594 29484 29484
query68
query69 461 346 312 312
query70 1031 1021 957 957
query71 320 278 260 260
query72 2959 2703 2437 2437
query73 884 765 424 424
query74 5101 4967 4769 4769
query75 2711 2599 2292 2292
query76 2288 1153 788 788
query77 421 429 349 349
query78 12479 12302 11792 11792
query79 1418 1009 768 768
query80 649 543 451 451
query81 457 280 243 243
query82 1389 164 124 124
query83 357 308 255 255
query84 265 139 113 113
query85 908 540 471 471
query86 410 347 321 321
query87 3441 3376 3238 3238
query88 3650 2773 2799 2773
query89 439 392 344 344
query90 1887 188 183 183
query91 185 173 140 140
query92 81 77 80 77
query93 1505 1593 932 932
query94 551 366 304 304
query95 681 402 447 402
query96 1125 838 331 331
query97 2760 2737 2638 2638
query98 233 228 229 228
query99 1155 1161 1016 1016
Total cold run time: 254645 ms
Total hot run time: 172100 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4581803983
# BE UT Coverage Report
Increment line coverage `100.00% (91/91)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2cb8f4c0b242cf643c5cfca3f7af047058786467_2cb8f4c0b242cf643c5cfca3f7af047058786467/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2cb8f4c0b242cf643c5cfca3f7af047058786467_2cb8f4c0b242cf643c5cfca3f7af047058786467/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.03% (21021/38907) |
| Line Coverage | 37.57% (199288/530401) |
| Region Coverage | 33.85% (156092/461070) |
| Branch Coverage | 34.85% (67962/195027) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4517896279
<details>
<summary>TPC-DS: <b>Total hot run time: 168964 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3727adef8e63244af0c76a060090befb5fe9ffed, data reload: false
query5 4319 649 546 546
query6 353 222 207 207
query7 4232 581 303 303
query8 327 247 236 236
query9 8850 4005 3970 3970
query10 465 340 303 303
query11 5779 2422 2198 2198
query12 182 132 122 122
query13 1284 598 446 446
query14 5901 5362 5077 5077
query14_1 4344 4372 4387 4372
query15 216 207 185 185
query16 1018 464 435 435
query17 964 739 606 606
query18 2471 486 371 371
query19 221 204 184 184
query20 136 133 134 133
query21 211 140 121 121
query22 13547 13640 13323 13323
query23 17266 16419 16036 16036
query23_1 16187 16183 16216 16183
query24 7374 1739 1290 1290
query24_1 1288 1262 1292 1262
query25 560 467 401 401
query26 1304 343 171 171
query27 2712 548 340 340
query28 4451 1969 1923 1923
query29 1024 624 492 492
query30 304 240 200 200
query31 1098 1061 935 935
query32 87 74 72 72
query33 547 363 319 319
query34 1171 1108 629 629
query35 757 785 672 672
query36 1321 1346 1177 1177
query37 154 103 105 103
query38 3196 3173 3061 3061
query39 931 908 894 894
query39_1 875 877 888 877
query40 235 150 131 131
query41 68 64 63 63
query42 109 110 106 106
query43 315 327 278 278
query44
query45 211 195 196 195
query46 1077 1148 704 704
query47 2334 2277 2229 2229
query48 428 417 294 294
query49 638 498 396 396
query50 962 344 257 257
query51 4278 4271 4224 4224
query52 107 103 95 95
query53 262 276 203 203
query54 324 275 266 266
query55 92 95 88 88
query56 298 312 295 295
query57 1412 1408 1288 1288
query58 297 271 293 271
query59 1582 1618 1424 1424
query60 323 315 301 301
query61 154 153 152 152
query62 658 621 564 564
query63 247 203 209 203
query64 2404 794 631 631
query65
query66 1741 488 355 355
query67 30003 29911 29790 29790
query68
query69 469 354 304 304
query70 1042 992 996 992
query71 309 269 272 269
query72 2977 2702 2401 2401
query73 852 746 432 432
query74 5035 4913 4737 4737
query75 2655 2564 2235 2235
query76 2294 1131 797 797
query77 398 407 340 340
query78 12126 12048 11612 11612
query79 1367 992 739 739
query80 632 517 456 456
query81 450 275 232 232
query82 623 172 119 119
query83 351 272 250 250
query84 311 137 108 108
query85 871 636 449 449
query86 380 332 343 332
query87 3399 3361 3189 3189
query88 3477 2655 2617 2617
query89 427 380 343 343
query90 1920 189 184 184
query91 177 171 144 144
query92 83 82 78 78
query93 1473 1428 839 839
query94 544 350 310 310
query95 680 374 347 347
query96 988 840 342 342
query97 2691 2686 2607 2607
query98 238 230 224 224
query99 1107 1103 997 997
Total cold run time: 251304 ms
Total hot run time: 168964 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #63517:
URL: https://github.com/apache/doris/pull/63517#discussion_r3323838482
##########
be/src/util/jsonb_document.h:
##########
@@ -363,19 +418,51 @@ struct leg_info {
Review Comment:
Thanks, fixed in `2cb8f4c0b24`. `has_escapes` is now reset at the start of each parsed path leg, so a following unescaped member no longer calls `remove_escapes()` and overwrites the next delimiter. I also added `jsonb_path_escaped_member_round_trips_with_following_legs` to cover `$."a\\nb".c.d`-style nested round trip with trailing legs.
Validation:
- `build-support/clang-format.sh`; `build-support/check-format.sh`; `git diff --check`
- `DORIS_HOME=$PWD ninja -C be/ut_build_ASAN test/CMakeFiles/doris_be_test.dir/core/jsonb/jsonb_document_test.cpp.o`
- standalone JsonbPath nested escaped-member repro compiled and ran successfully
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4531290568
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4551031175
<details>
<summary>TPC-DS: <b>Total hot run time: 171025 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f9e924eff56aaa9cfcb214c84235c7c08cff2750, data reload: false
query5 4321 669 523 523
query6 333 225 205 205
query7 4339 580 306 306
query8 324 236 218 218
query9 8805 4069 4045 4045
query10 464 352 295 295
query11 5759 2657 2196 2196
query12 185 130 124 124
query13 1271 643 459 459
query14 6150 5487 5172 5172
query14_1 4511 4506 4448 4448
query15 217 208 190 190
query16 1028 462 420 420
query17 1191 741 612 612
query18 2570 490 368 368
query19 236 240 163 163
query20 140 127 130 127
query21 220 136 120 120
query22 13632 13599 13379 13379
query23 17334 16620 16249 16249
query23_1 16331 16427 16534 16427
query24 7607 1804 1330 1330
query24_1 1292 1326 1335 1326
query25 547 481 427 427
query26 1303 343 173 173
query27 2725 571 344 344
query28 4423 2000 1959 1959
query29 998 637 493 493
query30 304 238 195 195
query31 1136 1098 950 950
query32 89 78 76 76
query33 542 351 304 304
query34 1194 1158 637 637
query35 775 805 699 699
query36 1402 1407 1236 1236
query37 158 108 94 94
query38 3203 3147 3068 3068
query39 924 933 910 910
query39_1 901 896 875 875
query40 229 148 123 123
query41 65 63 66 63
query42 108 112 113 112
query43 331 329 289 289
query44
query45 213 206 197 197
query46 1071 1220 745 745
query47 2345 2310 2252 2252
query48 398 410 296 296
query49 636 498 393 393
query50 1005 344 259 259
query51 4371 4353 4227 4227
query52 104 107 93 93
query53 257 298 208 208
query54 312 274 257 257
query55 98 90 85 85
query56 302 309 322 309
query57 1455 1409 1310 1310
query58 312 286 280 280
query59 1599 1698 1407 1407
query60 330 340 326 326
query61 180 182 180 180
query62 701 653 592 592
query63 240 207 207 207
query64 2462 849 678 678
query65
query66 1732 488 378 378
query67 29738 29020 29525 29020
query68
query69 475 354 307 307
query70 1044 1007 1042 1007
query71 317 328 267 267
query72 3024 2771 2396 2396
query73 842 754 436 436
query74 5144 4990 4786 4786
query75 2723 2617 2253 2253
query76 2261 1160 767 767
query77 407 404 322 322
query78 12320 12516 11785 11785
query79 1478 1056 775 775
query80 1348 559 468 468
query81 524 281 240 240
query82 1112 160 123 123
query83 321 279 247 247
query84 271 143 117 117
query85 914 567 455 455
query86 473 350 340 340
query87 3430 3405 3230 3230
query88 3654 2714 2740 2714
query89 453 388 345 345
query90 1907 186 177 177
query91 181 174 142 142
query92 81 79 78 78
query93 1564 1384 853 853
query94 712 355 314 314
query95 664 479 346 346
query96 1020 755 352 352
query97 2740 2732 2608 2608
query98 239 228 228 228
query99 1179 1155 1030 1030
Total cold run time: 255325 ms
Total hot run time: 171025 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4531022457
# BE Regression && UT Coverage Report
Increment line coverage `68.89% (62/90)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63517_3727adef8e63244af0c76a060090befb5fe9ffed_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63517_3727adef8e63244af0c76a060090befb5fe9ffed_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.97% (27279/37903) |
| Line Coverage | 55.30% (291228/526612) |
| Region Coverage | 52.19% (241865/463424) |
| Branch Coverage | 53.59% (104452/194895) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4517820707
<details>
<summary>TPC-H: <b>Total hot run time: 31174 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3727adef8e63244af0c76a060090befb5fe9ffed, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17715 3895 3892 3892
q2 q3 10764 1409 838 838
q4 4696 475 349 349
q5 7583 2295 2093 2093
q6 249 174 137 137
q7 950 793 650 650
q8 9366 1698 1660 1660
q9 5171 4915 4883 4883
q10 6397 2082 1779 1779
q11 441 278 245 245
q12 641 413 289 289
q13 18121 3494 2757 2757
q14 265 257 233 233
q15 q16 825 780 710 710
q17 953 962 978 962
q18 6915 5664 5553 5553
q19 1194 1258 1082 1082
q20 549 408 254 254
q21 5580 2593 2497 2497
q22 435 362 311 311
Total cold run time: 98810 ms
Total hot run time: 31174 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4181 4115 4114 4114
q2 q3 4519 4948 4349 4349
q4 2088 2208 1398 1398
q5 4408 4287 4284 4284
q6 229 174 130 130
q7 1771 2199 1702 1702
q8 2548 2234 2139 2139
q9 7957 8146 7825 7825
q10 4525 4524 4127 4127
q11 565 412 390 390
q12 718 726 517 517
q13 3324 3552 3036 3036
q14 295 325 274 274
q15 q16 725 726 653 653
q17 1350 1321 1307 1307
q18 8238 7252 7277 7252
q19 1200 1122 1097 1097
q20 2212 2202 1922 1922
q21 5276 4622 4469 4469
q22 534 456 414 414
Total cold run time: 56663 ms
Total hot run time: 51399 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63517:
URL: https://github.com/apache/doris/pull/63517#discussion_r3321622374
##########
be/src/util/jsonb_document.h:
##########
@@ -363,19 +418,51 @@ struct leg_info {
Review Comment:
`has_escapes` is never reset between path legs, so once an escaped member is parsed every later member also calls `remove_escapes()`. For a nested path produced by this change, for example a search result like `$."a\nb".c.d`, parsing the first member sets the flag; parsing the following unescaped `c` then calls `remove_escapes()`, which writes `\0` at `leg_ptr[new_len]` and overwrites the next `.` delimiter. The next loop in `seek()` sees the corrupted delimiter and the round-trip fails; if the following member is the final leg this write is at the end of the caller's path slice. Please clear `has_escapes` for each leg (for example alongside `clear_leg_ptr()`/`clear_leg_len()`, or after removal) and add a nested JSON_SEARCH -> JSON_EXTRACT / JsonbPath round-trip test so escaped member paths with trailing legs are covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4514884834
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4516395450
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4516727857
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4531290416
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4551053303
OpenCode automated review failed and did not complete.
Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/26486006863
Please inspect the workflow logs and rerun the review after the underlying issue is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4550564975
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg merged PR #63517:
URL: https://github.com/apache/doris/pull/63517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4574126299
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4581936087
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (91/91)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63517_2cb8f4c0b242cf643c5cfca3f7af047058786467_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63517_2cb8f4c0b242cf643c5cfca3f7af047058786467_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.84% (28131/38096) |
| Line Coverage | 57.84% (305986/529023) |
| Region Coverage | 54.95% (255760/465470) |
| Branch Coverage | 56.51% (110614/195753) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4581351499
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4583431315
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (91/91)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63517_2cb8f4c0b242cf643c5cfca3f7af047058786467_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63517_2cb8f4c0b242cf643c5cfca3f7af047058786467_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.86% (28136/38096) |
| Line Coverage | 57.84% (305997/529023) |
| Region Coverage | 54.97% (255861/465470) |
| Branch Coverage | 56.52% (110638/195753) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4551449634
# BE UT Coverage Report
Increment line coverage `100.00% (90/90)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/f9e924eff56aaa9cfcb214c84235c7c08cff2750_f9e924eff56aaa9cfcb214c84235c7c08cff2750/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/f9e924eff56aaa9cfcb214c84235c7c08cff2750_f9e924eff56aaa9cfcb214c84235c7c08cff2750/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.87% (20916/38829) |
| Line Coverage | 37.43% (198100/529301) |
| Region Coverage | 33.73% (155269/460371) |
| Branch Coverage | 34.71% (67575/194663) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4551667137
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (90/90)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63517_f9e924eff56aaa9cfcb214c84235c7c08cff2750_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63517_f9e924eff56aaa9cfcb214c84235c7c08cff2750_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.72% (28036/38029) |
| Line Coverage | 57.68% (304521/527945) |
| Region Coverage | 54.82% (254809/464794) |
| Branch Coverage | 56.27% (109952/195389) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4514884845
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4550980950
<details>
<summary>TPC-H: <b>Total hot run time: 31336 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f9e924eff56aaa9cfcb214c84235c7c08cff2750, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17743 4106 4049 4049
q2 q3 10729 1450 820 820
q4 4729 475 344 344
q5 8512 2312 2097 2097
q6 385 179 136 136
q7 973 792 630 630
q8 9402 1864 1647 1647
q9 7035 4979 4995 4979
q10 6468 2220 1887 1887
q11 439 287 242 242
q12 690 445 295 295
q13 18287 3349 2793 2793
q14 263 258 239 239
q15 q16 819 789 709 709
q17 1017 994 936 936
q18 6936 5910 5497 5497
q19 1213 1262 1032 1032
q20 510 407 259 259
q21 5705 2644 2441 2441
q22 428 360 304 304
Total cold run time: 102283 ms
Total hot run time: 31336 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4344 4282 4250 4250
q2 q3 4504 4906 4371 4371
q4 2082 2191 1397 1397
q5 4405 4278 5027 4278
q6 258 198 145 145
q7 2002 1778 1588 1588
q8 2466 2186 2125 2125
q9 8080 8035 7963 7963
q10 5064 4769 4299 4299
q11 555 415 375 375
q12 770 757 557 557
q13 3289 3703 2990 2990
q14 304 315 281 281
q15 q16 735 734 652 652
q17 1352 1375 1330 1330
q18 7887 7260 6776 6776
q19 1165 1115 1139 1115
q20 2215 2235 1950 1950
q21 5286 4541 4411 4411
q22 512 473 405 405
Total cold run time: 57275 ms
Total hot run time: 51258 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4516349858
Added regression coverage for JSON_SEARCH path escaping and refreshed json_search.out.
Validation:
- doris-local-regression --offset 10000 --network 10.26.20.3/24 run -d query_p0/sql_functions/json_functions -s test_json_search -forceGenOut
- doris-local-regression --offset 10000 --network 10.26.20.3/24 run -d query_p0/sql_functions/json_functions -s test_json_search
- git diff --check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4560682677
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (90/90)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63517_f9e924eff56aaa9cfcb214c84235c7c08cff2750_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63517_f9e924eff56aaa9cfcb214c84235c7c08cff2750_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.99% (27378/38029) |
| Line Coverage | 55.36% (292276/527945) |
| Region Coverage | 52.32% (243191/464794) |
| Branch Coverage | 53.60% (104719/195389) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Escape JSONB path member control characters [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #63517:
URL: https://github.com/apache/doris/pull/63517#issuecomment-4569667040
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org