You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "sollhui (via GitHub)" <gi...@apache.org> on 2026/10/21 08:09:36 UTC
[PR] [perf](fs) Push down Azure glob listing prefixes [doris]
sollhui opened a new pull request, #69383:
URL: https://github.com/apache/doris/pull/69383
### What problem does this PR solve?
Related PR: #64941, #64684
Azure glob listing scans only the longest static prefix. For example, `date=2025-{0[3-9],1[0-2]}-01/*` scans everything under `date=2025-`, including unrelated dates.
Extract the object-storage glob helpers into `ObjectStorageGlob` and use them for Azure and S3-compatible listing. Bounded alternatives become sorted, non-overlapping remote list prefixes; unsafe or excessive expansion falls back to a conservative prefix and still applies the complete matcher. Numeric ranges retain zero padding and can match large ranges without enumerating every value. Azure limits and resume cursors continue across expanded prefixes, and object-storage cursors consistently use unsigned UTF-8 order. S3 directory buckets retain slash-terminated prefixes and reject unsupported key-based pagination.
This reapplies the final changes from #64941 to current master (`cca2231716264a15cbc884fbeac96eef4eafd66b`). It adapts the S3 changes to `fe-filesystem-s3-base`, retains the existing SPI hooks and notification matcher, merges the offset test into master's existing tests, and preserves master's escaped/nested glob and suffix-sensitive prefix behavior. Hadoop's `GlobPattern` expansion remains unchanged.
### Release note
Reduce Azure glob listing scans for bounded patterns. Correct object-storage numeric-range and UTF-8 cursor handling. Reject limited/resumed S3 directory-bucket glob listing because it cannot support key-based pagination.
### Check List (For Author)
- Test:
- [x] Unit Test: ported and extended coverage for prefix pushdown, fallback, numeric ranges, padding, escaping, nesting, limits, UTF-8 cursors, and directory buckets.
- Local validation: Checkstyle passed for `fe-filesystem-spi`, `fe-filesystem-azure`, `fe-filesystem-s3-base`, and `fe-core`; `git diff --check` passed.
- Unit execution is pending CI. Attempted `bash run-fe-ut.sh --run ObjectStorageGlobTest,AzureFileSystemTest,S3FileSystemTest,S3CompatibleFileSystemTest,S3SourceOffsetProviderTest`, but the standard script stopped at the JDK check: this host has no configured JDK 17. No unit-test success is claimed.
- Behavior changed:
- [x] Yes. Azure pushes down expanded prefixes; object-storage ranges preserve padding and large-range semantics; cursor comparisons use UTF-8 order; directory-bucket key pagination fails explicitly.
- Does this need documentation?
- [x] No. No new configuration or syntax.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](storage) Prune Azure glob listing with expanded prefixes [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui merged PR #69383:
URL: https://github.com/apache/doris/pull/69383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6317272170
<details>
<summary>TPC-H: <b>Total hot run time: 28854 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit edfc98a8389b23f617a42efa9758414fd9c89c02, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17603 5152 5282 5152
q2 2102 323 259 259
q3 10214 1406 837 837
q4 4683 470 340 340
q5 7432 804 533 533
q6 191 187 159 159
q7 761 801 608 608
q8 9967 1334 1075 1075
q9 5655 4236 4222 4222
q10 6875 1303 1029 1029
q11 436 248 227 227
q12 708 397 278 278
q13 18247 3691 3038 3038
q14 279 281 252 252
q15 q16 777 716 663 663
q17 2796 765 590 590
q18 7061 6210 6134 6134
q19 1794 978 608 608
q20 409 346 227 227
q21 5793 2329 2486 2329
q22 394 331 294 294
Total cold run time: 104177 ms
Total hot run time: 28854 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5993 5975 5896 5896
q2 766 605 595 595
q3 5594 5658 5232 5232
q4 2022 2187 1566 1566
q5 6028 5709 5760 5709
q6 249 192 149 149
q7 2246 2034 1826 1826
q8 2976 2624 2670 2624
q9 7537 7511 7496 7496
q10 3674 3598 3177 3177
q11 579 440 414 414
q12 714 675 511 511
q13 3248 3574 3051 3051
q14 293 292 265 265
q15 q16 668 679 607 607
q17 7751 7130 6975 6975
q18 13035 12224 13023 12224
q19 918 794 817 794
q20 2154 2164 1944 1944
q21 5692 4798 4820 4798
q22 509 457 420 420
Total cold run time: 72646 ms
Total hot run time: 66273 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Push down Azure glob listing prefixes [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6311280280
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6336053547
# FE Regression Coverage Report
Increment line coverage ` 9.52% (6/63)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69383_32dfb1c24ad3b39d86581514e237f909738c3aa5_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69383_32dfb1c24ad3b39d86581514e237f909738c3aa5_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6317477437
<details>
<summary>TPC-DS: <b>Total hot run time: 150455 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit edfc98a8389b23f617a42efa9758414fd9c89c02, data reload: false
query5 4312 593 452 452
query6 427 204 189 189
query7 4803 440 220 220
query8 314 168 155 155
query9 8786 3063 3070 3063
query10 487 305 260 260
query11 6035 3864 3566 3566
query12 149 93 88 88
query13 1246 443 321 321
query14 6054 3938 3636 3636
query14_1 3380 3468 3415 3415
query15 209 210 186 186
query16 960 481 419 419
query17 864 665 526 526
query18 2432 419 312 312
query19 194 167 138 138
query20 87 87 85 85
query21 209 130 111 111
query22 12987 12977 12834 12834
query23 13417 12900 12430 12430
query23_1 12593 12490 12616 12490
query24 7412 817 462 462
query24_1 479 473 466 466
query25 561 425 368 368
query26 1258 259 138 138
query27 2764 444 277 277
query28 4639 1929 1917 1917
query29 1555 602 456 456
query30 298 259 179 179
query31 1010 928 799 799
query32 146 91 86 86
query33 508 292 219 219
query34 897 842 499 499
query35 730 744 652 652
query36 824 813 751 751
query37 116 94 79 79
query38 1773 1759 1642 1642
query39 796 789 734 734
query39_1 721 722 750 722
query40 215 111 91 91
query41 64 63 62 62
query42 82 79 79 79
query43 341 358 309 309
query44 1303 677 683 677
query45 179 175 167 167
query46 816 968 554 554
query47 2854 3045 2616 2616
query48 298 301 215 215
query49 561 392 293 293
query50 702 285 217 217
query51 10244 10532 10490 10490
query52 81 76 70 70
query53 185 205 159 159
query54 249 188 198 188
query55 76 67 64 64
query56 237 220 202 202
query57 1628 1624 1585 1585
query58 275 257 252 252
query59 2109 2144 1996 1996
query60 292 247 253 247
query61 162 148 145 145
query62 390 328 289 289
query63 184 156 154 154
query64 2708 950 746 746
query65 3346 3287 3261 3261
query66 1765 412 298 298
query67 20082 20132 19935 19935
query68 2973 984 596 596
query69 371 281 248 248
query70 845 794 754 754
query71 290 233 215 215
query72 2641 2730 2277 2277
query73 515 517 289 289
query74 4623 4536 4329 4329
query75 2305 2288 1893 1893
query76 2361 999 580 580
query77 333 377 274 274
query78 9908 9640 9198 9198
query79 1012 841 534 534
query80 539 412 332 332
query81 514 313 276 276
query82 576 132 109 109
query83 308 201 183 183
query84 307 120 104 104
query85 833 435 370 370
query86 321 238 233 233
query87 1995 1983 1799 1799
query88 3642 2646 2624 2624
query89 317 280 239 239
query90 1879 164 166 164
query91 161 152 127 127
query92 99 83 91 83
query93 979 984 564 564
query94 442 307 286 286
query95 577 320 357 320
query96 663 485 232 232
query97 2480 2482 2322 2322
query98 169 152 152 152
query99 773 765 653 653
Total cold run time: 233353 ms
Total hot run time: 150455 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6334208871
# FE UT Coverage Report
Increment line coverage ` 100.00% (8/8)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69383_32dfb1c24ad3b39d86581514e237f909738c3aa5/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69383_32dfb1c24ad3b39d86581514e237f909738c3aa5/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6331544614
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6333483423
<details>
<summary>TPC-H: <b>Total hot run time: 28324 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32dfb1c24ad3b39d86581514e237f909738c3aa5, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17621 5168 5065 5065
q2 2113 319 251 251
q3 10235 1352 829 829
q4 4674 457 332 332
q5 7425 786 534 534
q6 183 185 150 150
q7 752 788 597 597
q8 9252 1348 1105 1105
q9 5529 4212 4197 4197
q10 6833 1288 1049 1049
q11 443 252 229 229
q12 640 393 281 281
q13 18080 3561 2939 2939
q14 276 271 248 248
q15 q16 711 730 654 654
q17 1171 743 566 566
q18 6927 6160 6093 6093
q19 1102 973 604 604
q20 386 346 221 221
q21 4427 2134 2301 2134
q22 341 295 246 246
Total cold run time: 99121 ms
Total hot run time: 28324 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5618 5556 5606 5556
q2 680 567 534 534
q3 5221 5341 4940 4940
q4 1986 2118 1550 1550
q5 5286 5165 5456 5165
q6 251 190 148 148
q7 2383 2039 1812 1812
q8 3015 2670 2676 2670
q9 8007 7991 7983 7983
q10 3747 3674 3254 3254
q11 615 592 429 429
q12 713 716 530 530
q13 3461 3846 3239 3239
q14 315 324 300 300
q15 q16 696 709 633 633
q17 8373 7552 6931 6931
q18 13026 12193 12830 12193
q19 897 817 768 768
q20 2149 2125 1901 1901
q21 5679 4605 4702 4605
q22 519 464 414 414
Total cold run time: 72637 ms
Total hot run time: 65555 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6314732222
# FE UT Coverage Report
Increment line coverage ` 50.00% (1/2)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69383_edfc98a8389b23f617a42efa9758414fd9c89c02/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69383_edfc98a8389b23f617a42efa9758414fd9c89c02/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6333677244
<details>
<summary>ClickBench: <b>Total hot run time: 24.22 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 32dfb1c24ad3b39d86581514e237f909738c3aa5, data reload: false
query1 0.01 0.01 0.00
query2 0.09 0.05 0.05
query3 0.26 0.08 0.08
query4 1.61 0.10 0.09
query5 0.26 0.24 0.24
query6 1.17 0.67 0.66
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.39 0.34 0.33
query10 0.60 0.56 0.58
query11 0.19 0.14 0.14
query12 0.18 0.14 0.14
query13 0.49 0.49 0.48
query14 1.02 1.03 1.03
query15 0.62 0.60 0.60
query16 0.35 0.35 0.33
query17 1.16 1.15 1.11
query18 0.20 0.18 0.20
query19 2.11 1.99 1.93
query20 0.02 0.01 0.01
query21 15.48 0.20 0.13
query22 4.88 0.05 0.05
query23 16.29 0.30 0.12
query24 3.04 0.54 0.46
query25 0.11 0.04 0.06
query26 1.53 0.21 0.15
query27 0.06 0.03 0.04
query28 3.90 0.67 0.36
query29 12.52 4.01 3.23
query30 0.28 0.13 0.09
query31 2.77 0.58 0.33
query32 3.23 0.59 0.50
query33 3.20 3.25 3.22
query34 15.42 4.01 3.40
query35 3.39 3.33 3.33
query36 0.55 0.43 0.43
query37 0.09 0.06 0.06
query38 0.05 0.04 0.03
query39 0.04 0.03 0.04
query40 0.17 0.16 0.13
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.04
Total cold run time: 97.99 s
Total hot run time: 24.22 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6331548941
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6333611891
<details>
<summary>TPC-DS: <b>Total hot run time: 150979 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32dfb1c24ad3b39d86581514e237f909738c3aa5, data reload: false
query5 4294 584 450 450
query6 425 204 194 194
query7 4794 441 230 230
query8 333 182 148 148
query9 8770 3092 3104 3092
query10 472 309 257 257
query11 6011 3882 3570 3570
query12 151 96 89 89
query13 1286 462 321 321
query14 6112 3955 3644 3644
query14_1 3379 3434 3438 3434
query15 203 204 182 182
query16 941 445 405 405
query17 845 650 535 535
query18 2409 437 315 315
query19 193 165 134 134
query20 85 83 83 83
query21 208 129 108 108
query22 12855 12983 12787 12787
query23 13470 12837 12390 12390
query23_1 12584 12587 12495 12495
query24 7380 817 464 464
query24_1 481 461 472 461
query25 531 421 357 357
query26 1259 258 143 143
query27 2764 437 271 271
query28 4634 1959 1972 1959
query29 1571 598 448 448
query30 292 217 189 189
query31 996 931 826 826
query32 147 95 89 89
query33 518 299 254 254
query34 930 849 490 490
query35 758 761 663 663
query36 812 789 732 732
query37 132 94 78 78
query38 1775 1708 1676 1676
query39 788 782 747 747
query39_1 748 722 742 722
query40 221 119 98 98
query41 71 67 67 67
query42 86 87 81 81
query43 343 371 317 317
query44 1314 689 685 685
query45 179 176 162 162
query46 851 957 564 564
query47 2691 3044 2903 2903
query48 300 316 233 233
query49 600 408 346 346
query50 698 283 210 210
query51 10210 10640 10514 10514
query52 80 83 69 69
query53 182 206 151 151
query54 236 205 211 205
query55 68 67 70 67
query56 243 225 200 200
query57 1628 1640 1585 1585
query58 284 262 251 251
query59 2097 2122 1912 1912
query60 287 240 235 235
query61 148 141 144 141
query62 378 352 304 304
query63 192 153 157 153
query64 2740 936 794 794
query65 3341 3270 3270 3270
query66 1785 418 311 311
query67 20367 19995 20027 19995
query68 3054 999 621 621
query69 378 301 253 253
query70 837 762 766 762
query71 283 238 210 210
query72 2767 2571 2274 2274
query73 509 527 299 299
query74 4593 4525 4328 4328
query75 2337 2313 1915 1915
query76 2321 1015 598 598
query77 350 355 271 271
query78 9829 9707 9137 9137
query79 983 856 531 531
query80 1226 434 333 333
query81 601 317 268 268
query82 582 131 107 107
query83 302 202 185 185
query84 311 130 101 101
query85 861 425 363 363
query86 392 242 228 228
query87 2007 1959 1779 1779
query88 3712 2677 2661 2661
query89 340 292 243 243
query90 1860 168 168 168
query91 162 148 127 127
query92 97 89 86 86
query93 1122 1008 578 578
query94 637 295 280 280
query95 576 320 300 300
query96 664 485 234 234
query97 2460 2414 2350 2350
query98 160 153 149 149
query99 766 749 637 637
Total cold run time: 234728 ms
Total hot run time: 150979 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Push down Azure glob listing prefixes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6311235739
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Push down Azure glob listing prefixes [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6311271713
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69383:
URL: https://github.com/apache/doris/pull/69383#issuecomment-6317581368
<details>
<summary>ClickBench: <b>Total hot run time: 24.2 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit edfc98a8389b23f617a42efa9758414fd9c89c02, data reload: false
query1 0.01 0.01 0.01
query2 0.10 0.05 0.05
query3 0.26 0.08 0.09
query4 1.62 0.11 0.10
query5 0.27 0.24 0.24
query6 1.16 0.69 0.68
query7 0.04 0.00 0.01
query8 0.06 0.04 0.03
query9 0.42 0.34 0.34
query10 0.59 0.57 0.64
query11 0.21 0.14 0.14
query12 0.18 0.14 0.15
query13 0.47 0.50 0.51
query14 1.01 1.01 1.04
query15 0.64 0.61 0.60
query16 0.34 0.34 0.36
query17 1.15 1.12 1.15
query18 0.19 0.18 0.18
query19 2.04 2.00 1.92
query20 0.02 0.02 0.01
query21 15.48 0.19 0.13
query22 4.97 0.05 0.06
query23 16.25 0.32 0.13
query24 3.16 0.63 0.34
query25 0.11 0.05 0.06
query26 1.12 0.21 0.15
query27 0.05 0.06 0.05
query28 3.17 0.79 0.36
query29 12.51 4.03 3.22
query30 0.28 0.13 0.10
query31 2.77 0.54 0.32
query32 3.21 0.59 0.50
query33 3.25 3.19 3.22
query34 15.28 4.01 3.40
query35 3.36 3.36 3.37
query36 0.56 0.46 0.42
query37 0.10 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.15 0.15
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.05 0.03 0.03
Total cold run time: 96.84 s
Total hot run time: 24.2 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [perf](fs) Prune Azure glob listing with expanded prefixes [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69383:
URL: https://github.com/apache/doris/pull/69383#discussion_r4334770927
##########
fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystem.java:
##########
@@ -397,51 +398,62 @@ public GlobListing globListWithLimit(Location path, String startAfter, long maxB
// base = uri with the key portion stripped, preserving the original scheme/host syntax.
String base = uri.substring(0, uri.length() - keyPattern.length());
- Pattern matcher = Pattern.compile(globToRegex(keyPattern));
- String listKeyPrefix = longestNonGlobPrefix(keyPattern);
- String listPrefixUri = base + listKeyPrefix;
+ String expandedKeyPattern = ObjectStorageGlob.expandNumericRanges(keyPattern);
+ Pattern matcher = Pattern.compile(globToRegex(expandedKeyPattern));
+ String listKeyPrefix = longestNonGlobPrefix(expandedKeyPattern);
+ List<String> listKeyPrefixes = ObjectStorageGlob.expandedGlobListPrefixes(expandedKeyPattern);
Review Comment:
[P2] Fall back when class-range endpoints contain escapes
For `data/[\a-z]/*/file.csv`, the new prefix expander consumes `\a` as a complete alternative and emits only `data/-/`, `data/a/`, and `data/z/`. The full matcher emits `[\Qa\E-z]`, an ordinary `a-z` range, so it accepts `data/m/sub/file.csv`, but none of the remote LIST calls can return that blob. Escaped upper endpoints such as `[a-\c]` have the same mismatch. Azure previously listed the conservative `data/` prefix and found these matches; importing the existing S3 expander introduces silent omissions here. Parse escaped range endpoints consistently with the matcher, or retain a conservative prefix for these classes.
##########
fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystem.java:
##########
@@ -397,51 +398,62 @@ public GlobListing globListWithLimit(Location path, String startAfter, long maxB
// base = uri with the key portion stripped, preserving the original scheme/host syntax.
String base = uri.substring(0, uri.length() - keyPattern.length());
- Pattern matcher = Pattern.compile(globToRegex(keyPattern));
- String listKeyPrefix = longestNonGlobPrefix(keyPattern);
- String listPrefixUri = base + listKeyPrefix;
+ String expandedKeyPattern = ObjectStorageGlob.expandNumericRanges(keyPattern);
+ Pattern matcher = Pattern.compile(globToRegex(expandedKeyPattern));
+ String listKeyPrefix = longestNonGlobPrefix(expandedKeyPattern);
+ List<String> listKeyPrefixes = ObjectStorageGlob.expandedGlobListPrefixes(expandedKeyPattern);
List<FileEntry> files = new ArrayList<>();
long totalSize = 0L;
- String maxFile = "";
- String continuationToken = null;
+ String nextMatchAfterLimit = "";
+ String lastMatchedKey = "";
boolean reachLimit = false;
- outer:
- do {
- RemoteObjects page = objStorage.listObjects(listPrefixUri, continuationToken);
- for (RemoteObject obj : page.getObjectList()) {
- String key = obj.getKey();
- if (key.endsWith(DIR_MARKER_SUFFIX)) {
- continue;
- }
- if (startAfter != null && !startAfter.isEmpty() && key.compareTo(startAfter) <= 0) {
- continue;
- }
- if (!matcher.matcher(key).matches()) {
- continue;
- }
- if ((maxFiles > 0 && files.size() >= maxFiles)
- || (maxBytes > 0 && totalSize >= maxBytes)) {
- maxFile = key;
- reachLimit = true;
- break outer;
+ for (String currentListKeyPrefix : listKeyPrefixes) {
+ String continuationToken = null;
+ String listPrefixUri = base + currentListKeyPrefix;
Review Comment:
[P2] Encode expanded prefixes before rebuilding the Azure URI
A valid glob such as `data/[$-&]/*/file.csv` expands to prefixes containing `$`, `%`, and `&`. This concatenation sends the generated `data/%/` through `AzureObjStorage.listObjects`, which reparses it with `AzureUri.parse` and `URLDecoder.decode`; the bare percent escape throws `IllegalArgumentException`, aborting the listing even if the container only has ordinary matching `$`/`&` keys. The previous conservative `data/` scan succeeded. Generated `+` and `#` prefixes are likewise changed into spaces or truncated. Encode the raw prefix while preserving path separators, or pass the raw prefix directly to the listing API. The storage mocks in the new tests bypass this second parse.
##########
fe/fe-filesystem/fe-filesystem-azure/src/main/java/org/apache/doris/filesystem/azure/AzureFileSystem.java:
##########
@@ -397,51 +398,62 @@ public GlobListing globListWithLimit(Location path, String startAfter, long maxB
// base = uri with the key portion stripped, preserving the original scheme/host syntax.
String base = uri.substring(0, uri.length() - keyPattern.length());
- Pattern matcher = Pattern.compile(globToRegex(keyPattern));
- String listKeyPrefix = longestNonGlobPrefix(keyPattern);
- String listPrefixUri = base + listKeyPrefix;
+ String expandedKeyPattern = ObjectStorageGlob.expandNumericRanges(keyPattern);
+ Pattern matcher = Pattern.compile(globToRegex(expandedKeyPattern));
+ String listKeyPrefix = longestNonGlobPrefix(expandedKeyPattern);
+ List<String> listKeyPrefixes = ObjectStorageGlob.expandedGlobListPrefixes(expandedKeyPattern);
List<FileEntry> files = new ArrayList<>();
long totalSize = 0L;
- String maxFile = "";
- String continuationToken = null;
+ String nextMatchAfterLimit = "";
+ String lastMatchedKey = "";
boolean reachLimit = false;
- outer:
- do {
- RemoteObjects page = objStorage.listObjects(listPrefixUri, continuationToken);
- for (RemoteObject obj : page.getObjectList()) {
- String key = obj.getKey();
- if (key.endsWith(DIR_MARKER_SUFFIX)) {
- continue;
- }
- if (startAfter != null && !startAfter.isEmpty() && key.compareTo(startAfter) <= 0) {
- continue;
- }
- if (!matcher.matcher(key).matches()) {
- continue;
- }
- if ((maxFiles > 0 && files.size() >= maxFiles)
- || (maxBytes > 0 && totalSize >= maxBytes)) {
- maxFile = key;
- reachLimit = true;
- break outer;
+ for (String currentListKeyPrefix : listKeyPrefixes) {
+ String continuationToken = null;
+ String listPrefixUri = base + currentListKeyPrefix;
+ do {
+ RemoteObjects page = objStorage.listObjects(listPrefixUri, continuationToken);
+ for (RemoteObject obj : page.getObjectList()) {
+ String key = obj.getKey();
+ if (key.endsWith(DIR_MARKER_SUFFIX)) {
+ continue;
+ }
+ if (startAfter != null && !startAfter.isEmpty()
+ && ObjectStorageGlob.compareUtf8Binary(key, startAfter) <= 0) {
Review Comment:
[P2] Apply resume cursors in the Azure endpoint listing order
The configurable Azure endpoint can use Microsoft's Azurite, whose [flat listing implementation](https://github.com/Azure/Azurite/blob/main/src/blob/persistence/LokiBlobMetadataStore.ts) sorts names with JavaScript string comparisons (UTF-16 ordinal). For `data/*.csv`, that order puts `data/😀.csv` before `data/Ａ.csv` (U+FF21). With `maxFiles=1`, the first call returns the emoji key; resuming from that returned key now discards the unconsumed fullwidth-A key because it sorts before the emoji in UTF-8. The former `String.compareTo` retained it. Keep prefix ordering, remote enumeration and cursor filtering consistent with the endpoint order, and cover a limited no-brace listing in ordinal order. This counterexample is established from the official emulator source; live Azure service ordering was not exercised.
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/ObjectStorageGlob.java:
##########
@@ -0,0 +1,725 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import java.math.BigInteger;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/**
+ * Provider-neutral glob helpers for object-storage keys.
+ */
+public final class ObjectStorageGlob {
+
+ private static final int MAX_EXPANDED_GLOB_LIST_PREFIXES = 256;
+ private static final Comparator<String> UTF8_BINARY_ORDER = ObjectStorageGlob::compareUtf8Binary;
+ private static final Pattern NUMERIC_RANGE_PATTERN = Pattern.compile("-?\\d+\\.\\.-?\\d+");
+ private static final Pattern NUMERIC_RANGE_ALTERNATIVE = Pattern.compile("(-?\\d+)\\.\\.(-?\\d+)");
+
+ private ObjectStorageGlob() {
+ }
+
+ /**
+ * Returns the longest key prefix that contains no glob metacharacters.
+ */
+ public static String longestNonGlobPrefix(String globPattern) {
+ int earliest = globPattern.length();
+ for (char c : new char[]{'*', '?', '[', '{', '\\'}) {
+ int idx = globPattern.indexOf(c);
+ if (idx >= 0 && idx < earliest) {
+ earliest = idx;
+ }
+ }
+ return globPattern.substring(0, earliest);
+ }
+
+ /**
+ * Returns object-store list prefixes that are safe to push down for a glob pattern.
+ *
+ * <p>Unlike {@link #longestNonGlobPrefix(String)}, this expands bounded glob constructs
+ * ({@code {...}} alternation and positive {@code [...]} character classes) before the first
+ * unbounded wildcard. For example,
+ * {@code date=2025-{0[3-9],1[0-2]}-01/mp_id=8/*} becomes concrete date prefixes instead
+ * of one broad {@code date=2025-} scan. If expansion would be too large or unsafe, it
+ * falls back to the conservative longest static prefix.
+ */
+ public static List<String> expandedGlobListPrefixes(String globPattern) {
+ List<String> prefixes = expandGlobListPrefixes(globPattern, true);
+ return prefixes == null ? List.of(longestNonGlobPrefix(globPattern)) : prefixes;
+ }
+
+ private static List<String> expandGlobListPrefixes(String globPattern, boolean allowPartialPrefix) {
+ List<String> prefixes = new ArrayList<>();
+ prefixes.add("");
+ int i = 0;
+ while (i < globPattern.length()) {
+ char c = globPattern.charAt(i);
+ if (c == '*' || c == '?') {
+ return allowPartialPrefix ? compactPrefixes(prefixes) : null;
+ }
+ if (c == '\\') {
+ if (i + 1 < globPattern.length()) {
+ appendLiteral(prefixes, globPattern.charAt(i + 1));
+ i += 2;
+ } else {
+ appendLiteral(prefixes, c);
+ i++;
+ }
+ continue;
+ }
+ if (c == '[') {
+ PrefixExpansion charClass = expandCharacterClass(globPattern, i);
+ if (charClass == null) {
+ return allowPartialPrefix ? compactPrefixes(prefixes) : null;
+ }
+ prefixes = appendAlternatives(prefixes, charClass.values);
+ if (prefixes == null) {
+ return null;
+ }
+ i = charClass.nextIndex;
+ continue;
+ }
+ if (c == '{') {
+ PrefixExpansion brace = expandBraceGroup(globPattern, i);
+ if (brace == null) {
+ return allowPartialPrefix ? compactPrefixes(prefixes) : null;
+ }
+ prefixes = appendAlternatives(prefixes, brace.values);
+ if (prefixes == null) {
+ return null;
+ }
+ i = brace.nextIndex;
+ continue;
+ }
+ appendLiteral(prefixes, c);
+ i++;
+ }
+ // Complete brace arms must retain distinct values until the surrounding suffix is appended.
+ return allowPartialPrefix ? compactPrefixes(prefixes) : prefixes;
+ }
+
+ private static void appendLiteral(List<String> prefixes, char c) {
+ for (int i = 0; i < prefixes.size(); i++) {
+ prefixes.set(i, prefixes.get(i) + c);
+ }
+ }
+
+ private static List<String> appendAlternatives(List<String> prefixes, List<String> alternatives) {
+ long expandedSize = (long) prefixes.size() * alternatives.size();
+ if (expandedSize > MAX_EXPANDED_GLOB_LIST_PREFIXES) {
+ return null;
+ }
+ List<String> expanded = new ArrayList<>((int) expandedSize);
+ for (String prefix : prefixes) {
+ for (String alternative : alternatives) {
+ expanded.add(prefix + alternative);
+ }
+ }
+ return expanded;
+ }
+
+ private static PrefixExpansion expandCharacterClass(String globPattern, int openIndex) {
+ int closeIndex = findClosingBracket(globPattern, openIndex);
+ if (closeIndex < 0 || closeIndex == openIndex + 1) {
+ return null;
+ }
+ int i = openIndex + 1;
+ char first = globPattern.charAt(i);
+ if (first == '!' || first == '^') {
+ return null;
+ }
+ if (containsSurrogate(globPattern, i, closeIndex)) {
+ return null;
+ }
+ List<String> values = new ArrayList<>();
+ while (i < closeIndex) {
+ char current = globPattern.charAt(i);
+ if (current == '\\') {
+ if (i + 1 >= closeIndex) {
+ return null;
+ }
+ values.add(String.valueOf(globPattern.charAt(i + 1)));
+ i += 2;
+ continue;
+ }
+ if (i + 2 < closeIndex && globPattern.charAt(i + 1) == '-') {
+ char rangeEnd = globPattern.charAt(i + 2);
+ int step = current <= rangeEnd ? 1 : -1;
+ for (char ch = current; step > 0 ? ch <= rangeEnd : ch >= rangeEnd; ch += step) {
+ values.add(String.valueOf(ch));
+ if (values.size() > MAX_EXPANDED_GLOB_LIST_PREFIXES) {
+ return null;
+ }
+ }
+ i += 3;
+ continue;
+ }
+ values.add(String.valueOf(current));
+ i++;
+ }
+ return new PrefixExpansion(values, closeIndex + 1);
+ }
+
+ private static int findClosingBracket(String globPattern, int openIndex) {
+ for (int i = openIndex + 1; i < globPattern.length(); i++) {
+ char c = globPattern.charAt(i);
+ if (c == '\\') {
+ i++;
+ continue;
+ }
+ if (c == ']') {
+ return i;
+ }
+ }
+ return -1;
+ }
+
+ private static boolean containsSurrogate(String text, int start, int end) {
+ for (int i = start; i < end; i++) {
+ if (Character.isSurrogate(text.charAt(i))) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private static PrefixExpansion expandBraceGroup(String globPattern, int openIndex) {
+ int closeIndex = findClosingBrace(globPattern, openIndex);
+ if (closeIndex < 0) {
+ return null;
+ }
+ List<String> alternatives = splitBraceAlternatives(
+ globPattern.substring(openIndex + 1, closeIndex));
+ if (alternatives.isEmpty()) {
+ return null;
+ }
+ List<String> values = new ArrayList<>();
+ for (String alternative : alternatives) {
+ if (containsNumericRange(alternative)) {
+ return null;
+ }
+ List<String> expandedAlternative = expandGlobListPrefixes(alternative, false);
+ if (expandedAlternative == null) {
+ return null;
+ }
+ values.addAll(expandedAlternative);
+ if (values.size() > MAX_EXPANDED_GLOB_LIST_PREFIXES) {
+ return null;
+ }
+ }
+ return new PrefixExpansion(values, closeIndex + 1);
+ }
+
+ private static int findClosingBrace(String globPattern, int openIndex) {
+ int depth = 0;
+ for (int i = openIndex; i < globPattern.length(); i++) {
+ char c = globPattern.charAt(i);
+ if (c == '\\') {
+ i++;
+ continue;
+ }
+ if (c == '[') {
+ int closeBracket = findClosingBracket(globPattern, i);
+ if (closeBracket >= 0) {
+ i = closeBracket;
+ continue;
+ }
+ }
+ if (c == '{') {
+ depth++;
+ } else if (c == '}') {
+ depth--;
+ if (depth == 0) {
+ return i;
+ }
+ }
+ }
+ return -1;
+ }
+
+ private static List<String> splitBraceAlternatives(String content) {
+ List<String> alternatives = new ArrayList<>();
+ int depth = 0;
+ int start = 0;
+ for (int i = 0; i < content.length(); i++) {
+ char c = content.charAt(i);
+ if (c == '\\') {
+ i++;
+ continue;
+ }
+ if (c == '[') {
+ int closeBracket = findClosingBracket(content, i);
+ if (closeBracket >= 0) {
+ i = closeBracket;
+ continue;
+ }
+ }
+ if (c == '{') {
+ depth++;
+ } else if (c == '}') {
+ if (depth == 0) {
+ return Collections.emptyList();
+ }
+ depth--;
+ } else if (c == ',' && depth == 0) {
+ alternatives.add(content.substring(start, i));
+ start = i + 1;
+ }
+ }
+ if (depth != 0) {
+ return Collections.emptyList();
+ }
+ alternatives.add(content.substring(start));
+ return alternatives;
+ }
+
+ private static List<String> compactPrefixes(List<String> prefixes) {
+ List<String> sorted = new ArrayList<>(prefixes);
+ sorted.sort(UTF8_BINARY_ORDER);
+ List<String> compact = new ArrayList<>();
+ for (String prefix : sorted) {
+ if (prefix.isEmpty()) {
+ return List.of("");
+ }
+ boolean covered = false;
+ for (String existing : compact) {
+ if (prefix.startsWith(existing)) {
+ covered = true;
+ break;
+ }
+ }
+ if (!covered) {
+ compact.add(prefix);
+ }
+ }
+ return compact;
+ }
+
+ /**
+ * Compares object keys by unsigned UTF-8 byte order, matching object-store listing order.
+ */
+ public static int compareUtf8Binary(String left, String right) {
+ byte[] leftBytes = left.getBytes(StandardCharsets.UTF_8);
+ byte[] rightBytes = right.getBytes(StandardCharsets.UTF_8);
+ int commonLength = Math.min(leftBytes.length, rightBytes.length);
+ for (int i = 0; i < commonLength; i++) {
+ int result = Integer.compare(
+ Byte.toUnsignedInt(leftBytes[i]), Byte.toUnsignedInt(rightBytes[i]));
+ if (result != 0) {
+ return result;
+ }
+ }
+ return Integer.compare(leftBytes.length, rightBytes.length);
+ }
+
+ /**
+ * Expands bounded {N..M} numeric ranges inside brace groups into comma-separated alternatives.
+ */
+ public static String expandNumericRanges(String pattern) {
+ StringBuilder expanded = new StringBuilder();
+ for (int i = 0; i < pattern.length(); i++) {
+ char c = pattern.charAt(i);
+ if (c == '\\') {
+ expanded.append(c);
+ if (i + 1 < pattern.length()) {
+ expanded.append(pattern.charAt(++i));
+ }
+ } else if (c == '[') {
+ int closeIndex = findClosingBracket(pattern, i);
+ if (closeIndex >= 0) {
+ expanded.append(pattern, i, closeIndex + 1);
+ i = closeIndex;
+ } else {
+ expanded.append(c);
+ }
+ } else if (c == '{') {
+ int closeIndex = findClosingBrace(pattern, i);
+ if (closeIndex >= 0) {
+ expanded.append('{').append(expandNumericBraceGroup(
+ pattern.substring(i + 1, closeIndex))).append('}');
+ i = closeIndex;
+ } else {
+ expanded.append(c);
+ }
+ } else {
+ expanded.append(c);
+ }
+ }
+ return expanded.toString();
+ }
+
+ private static String expandNumericBraceGroup(String content) {
+ List<String> segments = splitBraceAlternatives(content);
+ boolean isMixed = segments.size() > 1;
+ LinkedHashSet<String> values = new LinkedHashSet<>();
+ for (String segment : segments) {
+ Matcher range = numericRangeMatcher(segment, isMixed);
+ if (range == null) {
+ values.add(expandNumericRanges(segment));
+ } else {
+ long from;
+ long to;
+ try {
+ from = Long.parseLong(range.group(1));
+ to = Long.parseLong(range.group(2));
+ } catch (NumberFormatException e) {
+ return content;
+ }
+ long cardinality = numericRangeCardinality(from, to);
+ if (cardinality < 0 || cardinality > MAX_EXPANDED_GLOB_LIST_PREFIXES - values.size()) {
+ return content;
+ }
+ long step = from <= to ? 1L : -1L;
+ boolean zeroPad = hasLeadingZero(range.group(1)) || hasLeadingZero(range.group(2));
+ int width = Math.max(digitWidth(range.group(1)), digitWidth(range.group(2)));
+ for (long offset = 0; offset < cardinality; offset++) {
+ long value = step > 0 ? from + offset : from - offset;
+ values.add(formatRangeValue(value, width, zeroPad));
+ }
+ }
+ if (values.size() > MAX_EXPANDED_GLOB_LIST_PREFIXES) {
+ return content;
+ }
+ }
+ return String.join(",", values);
+ }
+
+ private static Matcher numericRangeMatcher(String alternative, boolean isMixed) {
+ Matcher matcher = NUMERIC_RANGE_ALTERNATIVE.matcher(isMixed ? alternative.trim() : alternative);
+ if (!matcher.matches() || (!isMixed
+ && (matcher.group(1).startsWith("-") || matcher.group(2).startsWith("-")))) {
+ return null;
+ }
+ return matcher;
+ }
+
+ private static boolean containsNumericRange(String text) {
+ return NUMERIC_RANGE_PATTERN.matcher(text).find();
+ }
+
+ private static long numericRangeCardinality(long from, long to) {
+ try {
+ long distance = from <= to
+ ? Math.subtractExact(to, from)
+ : Math.subtractExact(from, to);
+ return Math.addExact(distance, 1L);
+ } catch (ArithmeticException e) {
+ return -1L;
+ }
+ }
+
+ private static boolean hasLeadingZero(String value) {
+ String digits = value.startsWith("-") ? value.substring(1) : value;
+ return digits.length() > 1 && digits.charAt(0) == '0';
+ }
+
+ private static int digitWidth(String value) {
+ return value.startsWith("-") ? value.length() - 1 : value.length();
+ }
+
+ private static String formatRangeValue(long value, int width, boolean zeroPad) {
+ if (!zeroPad) {
+ return Long.toString(value);
+ }
+ if (value == Long.MIN_VALUE) {
Review Comment:
[P2] Preserve zero padding for the minimum long value
The early return skips the padding loop even when the requested width exceeds 19 digits. For the valid mixed range `data_{-09223372036854775808..-09223372036854775807,x}.csv`, both endpoints parse as longs, the cardinality is 2 and the digit width is 20, but normalization emits `-9223372036854775808` for the first arm and keeps the leading zero for the second. Consequently Azure/S3 listing and `compileGlobPattern` omit the correctly padded minimum key and accept its unpadded counterpart. The BigInteger regex fallback preserves this width, so behavior also differs between normalization paths. Derive magnitude digits without negating `Long.MIN_VALUE`, then apply the common padding logic to this value as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org