You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "jacktengg (via GitHub)" <gi...@apache.org> on 2026/10/13 15:18:22 UTC
[PR] [fix](be) Bound LEFT results for truncated UTF-8 input [doris]
jacktengg opened a new pull request, #69038:
URL: https://github.com/apache/doris/pull/69038
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: LEFT advances by the width declared by a UTF-8 lead byte even when the current row ends before that width. For adjacent rows containing E4B8 and 5A, LEFT(s, 1) can return E4B85A and persist the extra byte through CTAS or INSERT. Traverse using byte offsets and clamp the final slice to the current StringRef length before copying. LEFT and its STRLEFT alias now preserve the truncated bytes without including bytes from another row.
### Release note
Fix LEFT and STRLEFT returning extra bytes for truncated UTF-8 strings.
### Check List (For Author)
- Test: Unit Test / Regression test / Manual test
- ASAN BE unit tests: function_string_left_truncated_utf8_test and function_string_strleft_test; new test fails before the fix and passes after it
- Regression suite test_left_truncated_utf8: generate expected output and compare it in a separate run, including constant folding, CTAS and INSERT
- Original SQL reproduction returns E4B8 for LEFT(s, 1)
- ASAN BE build, GLIBC_2.17 compatibility gate, clang-format 16, build hygiene and clang-tidy checks passed
- Behavior changed: Yes; truncate LEFT/STRLEFT output at the current row boundary
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound LEFT results for truncated UTF-8 input [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69038:
URL: https://github.com/apache/doris/pull/69038#issuecomment-6155198413
<details>
<summary>ClickBench: <b>Total hot run time: 24.53 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9461e2475b6f6257f02654818bf8073dca556c8c, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.27 0.12 0.11
query4 1.61 0.13 0.13
query5 0.27 0.26 0.25
query6 1.18 0.68 0.68
query7 0.03 0.01 0.01
query8 0.05 0.04 0.04
query9 0.42 0.35 0.35
query10 0.57 0.57 0.58
query11 0.22 0.14 0.15
query12 0.19 0.14 0.14
query13 0.49 0.50 0.50
query14 0.97 0.98 0.96
query15 0.64 0.61 0.61
query16 0.36 0.35 0.36
query17 1.12 1.17 1.11
query18 0.22 0.21 0.21
query19 2.04 1.99 1.97
query20 0.02 0.02 0.01
query21 15.49 0.26 0.15
query22 4.49 0.05 0.05
query23 16.22 0.33 0.12
query24 3.33 0.59 0.38
query25 0.12 0.06 0.05
query26 1.07 0.25 0.15
query27 0.04 0.05 0.03
query28 1.74 0.60 0.42
query29 12.52 4.20 3.29
query30 0.28 0.14 0.13
query31 2.77 0.58 0.33
query32 3.22 0.61 0.51
query33 3.26 3.21 3.23
query34 15.65 4.00 3.39
query35 3.33 3.36 3.32
query36 0.57 0.47 0.43
query37 0.10 0.06 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.14 0.14
query41 0.09 0.03 0.03
query42 0.03 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 95.4 s
Total hot run time: 24.53 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound LEFT results for truncated UTF-8 input [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69038:
URL: https://github.com/apache/doris/pull/69038#issuecomment-6157402651
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69038_9461e2475b6f6257f02654818bf8073dca556c8c_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69038_9461e2475b6f6257f02654818bf8073dca556c8c_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.95% (35508/46146) |
| Line Coverage | 62.24% (402147/646112) |
| Region Coverage | 58.78% (340296/578912) |
| Branch Coverage | 59.62% (156519/262532) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound LEFT results for truncated UTF-8 input [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69038:
URL: https://github.com/apache/doris/pull/69038#issuecomment-6154953170
<details>
<summary>TPC-H: <b>Total hot run time: 29061 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9461e2475b6f6257f02654818bf8073dca556c8c, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17580 5661 5530 5530
q2 2106 311 276 276
q3 10245 1385 838 838
q4 4684 472 347 347
q5 7432 793 533 533
q6 188 185 157 157
q7 770 808 611 611
q8 9255 1344 1062 1062
q9 5704 4469 4460 4460
q10 6827 1293 1015 1015
q11 444 258 240 240
q12 630 412 287 287
q13 18055 3058 2368 2368
q14 265 264 243 243
q15 q16 745 724 673 673
q17 1187 740 593 593
q18 7102 6257 6194 6194
q19 1235 949 597 597
q20 388 337 220 220
q21 6060 2532 2595 2532
q22 415 367 285 285
Total cold run time: 101317 ms
Total hot run time: 29061 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6545 6455 6455 6455
q2 751 607 580 580
q3 5547 5746 5471 5471
q4 2293 2215 1595 1595
q5 5711 5736 5817 5736
q6 261 196 144 144
q7 2205 2028 1785 1785
q8 3136 2758 2781 2758
q9 8108 7716 7721 7716
q10 3667 3584 3245 3245
q11 619 432 410 410
q12 678 702 505 505
q13 2751 3101 2392 2392
q14 280 289 275 275
q15 q16 671 674 640 640
q17 7845 7056 6916 6916
q18 13080 12279 13079 12279
q19 949 836 843 836
q20 2193 2177 1943 1943
q21 5863 4739 4937 4739
q22 488 462 408 408
Total cold run time: 73641 ms
Total hot run time: 66828 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound LEFT results for truncated UTF-8 input [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69038:
URL: https://github.com/apache/doris/pull/69038#issuecomment-6156030430
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9461e2475b6f6257f02654818bf8073dca556c8c_9461e2475b6f6257f02654818bf8073dca556c8c/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9461e2475b6f6257f02654818bf8073dca556c8c_9461e2475b6f6257f02654818bf8073dca556c8c/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 65.87% (31405/47674) |
| Line Coverage | 50.61% (330699/653484) |
| Region Coverage | 45.95% (267125/581322) |
| Branch Coverage | 47.65% (125489/263343) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound LEFT results for truncated UTF-8 input [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69038:
URL: https://github.com/apache/doris/pull/69038#issuecomment-6148877258
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound LEFT results for truncated UTF-8 input [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #69038:
URL: https://github.com/apache/doris/pull/69038#issuecomment-6150213282
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound LEFT results for truncated UTF-8 input [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #69038:
URL: https://github.com/apache/doris/pull/69038#issuecomment-6148885789
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Bound LEFT results for truncated UTF-8 input [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69038:
URL: https://github.com/apache/doris/pull/69038#issuecomment-6155118595
<details>
<summary>TPC-DS: <b>Total hot run time: 152536 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9461e2475b6f6257f02654818bf8073dca556c8c, data reload: false
query5 4936 604 496 496
query6 433 219 192 192
query7 4809 450 229 229
query8 311 179 163 163
query9 8758 4037 4023 4023
query10 465 308 265 265
query11 5948 3426 3201 3201
query12 142 89 83 83
query13 1258 457 336 336
query14 6537 4819 4516 4516
query14_1 4271 4290 4286 4286
query15 216 210 184 184
query16 939 488 406 406
query17 886 665 561 561
query18 2393 408 304 304
query19 200 158 135 135
query20 79 78 83 78
query21 204 129 108 108
query22 13081 13020 12713 12713
query23 13001 12490 12076 12076
query23_1 12287 12181 12098 12098
query24 7075 841 457 457
query24_1 471 460 484 460
query25 517 390 351 351
query26 1248 251 133 133
query27 2777 440 268 268
query28 4646 1939 1940 1939
query29 1546 579 441 441
query30 312 236 181 181
query31 892 762 652 652
query32 144 102 85 85
query33 502 299 228 228
query34 1015 864 522 522
query35 747 759 650 650
query36 783 801 741 741
query37 134 106 94 94
query38 1802 1770 1659 1659
query39 719 695 699 695
query39_1 668 666 672 666
query40 225 115 100 100
query41 67 64 63 63
query42 84 81 82 81
query43 368 379 331 331
query44 1382 696 694 694
query45 190 187 165 165
query46 839 977 557 557
query47 2780 2959 2758 2758
query48 298 316 218 218
query49 576 393 287 287
query50 703 282 216 216
query51 10730 10606 10623 10606
query52 79 78 71 71
query53 193 214 152 152
query54 228 191 174 174
query55 74 72 62 62
query56 238 223 215 215
query57 1591 1596 1567 1567
query58 282 264 248 248
query59 2349 2419 2135 2135
query60 294 236 206 206
query61 163 149 147 147
query62 399 339 288 288
query63 188 168 171 168
query64 2744 925 855 855
query65 3444 3369 3406 3369
query66 1805 430 307 307
query67 20395 20157 20062 20062
query68 2959 1027 632 632
query69 389 300 256 256
query70 930 868 836 836
query71 294 231 224 224
query72 2790 2701 2628 2628
query73 527 534 300 300
query74 4548 4484 4289 4289
query75 2360 2319 1967 1967
query76 2312 1010 597 597
query77 373 463 307 307
query78 9069 9045 8509 8509
query79 976 911 528 528
query80 528 417 351 351
query81 539 327 272 272
query82 574 137 108 108
query83 314 197 185 185
query84 323 122 105 105
query85 819 424 373 373
query86 317 231 229 229
query87 2011 1968 1863 1863
query88 3661 2739 2699 2699
query89 337 307 252 252
query90 1896 193 186 186
query91 163 153 124 124
query92 98 83 83 83
query93 1018 1013 577 577
query94 450 311 279 279
query95 605 412 313 313
query96 658 530 235 235
query97 2446 2437 2302 2302
query98 160 146 138 138
query99 777 761 644 644
Total cold run time: 236296 ms
Total hot run time: 152536 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org