You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "dataroaring (via GitHub)" <gi...@apache.org> on 2026/02/01 20:44:42 UTC
[PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
dataroaring opened a new pull request, #60414:
URL: https://github.com/apache/doris/pull/60414
## Summary
- For S3 paths without wildcards (`*`, `?`, `[...]`), use HEAD requests instead of ListObjectsV2 to avoid requiring `s3:ListBucket` permission
- Brace patterns like `{1..10}` are expanded to concrete file paths and verified individually with HEAD requests
- This enables loading data from S3 when only `s3:GetObject` permission is granted
## Motivation
S3 `ListBucket` permission is often more restricted than `GetObject` in enterprise environments. When users specify exact file paths or deterministic patterns like `file{1..3}.csv`, listing is unnecessary since the file names can be determined from the input.
## Changes
| File | Description |
|------|-------------|
| `S3Util.java` | Added `isDeterministicPattern()` to detect paths without wildcards, and `expandBracePatterns()` to expand brace patterns to concrete paths |
| `S3ObjStorage.java` | Modified `globListInternal()` to use HEAD requests for deterministic paths |
| `S3UtilTest.java` | Added unit tests for new utility methods |
## Examples
| Path | Deterministic? | Behavior |
|------|----------------|----------|
| `s3://bucket/data/file.csv` | ✅ Yes | Single HEAD request |
| `s3://bucket/data/file{1..3}.csv` | ✅ Yes | 3 HEAD requests |
| `s3://bucket/data/*.csv` | ❌ No | Falls back to LIST |
## Test Plan
- [x] Added unit tests for `isDeterministicPattern()`
- [x] Added unit tests for `expandBracePatterns()`
- [ ] Manual testing with S3 TVF and Broker Load
🤖 Generated with [Claude Code](https://claude.ai/code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3862412150
<details>
<summary>TPC-H: <b>Total hot run time: 30500 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1e8a877cc6dbb43ae659d6c536e623e4b0f86721, data reload: false
------ Round 1 ----------------------------------
q1 17647 4440 4330 4330
q2 1998 346 230 230
q3 10179 1260 730 730
q4 10190 769 306 306
q5 7539 2209 1905 1905
q6 194 175 146 146
q7 879 744 593 593
q8 9296 1392 1155 1155
q9 4641 4640 4592 4592
q10 6830 1955 1580 1580
q11 515 322 310 310
q12 326 375 214 214
q13 17789 4022 3179 3179
q14 227 242 224 224
q15 868 821 788 788
q16 659 671 614 614
q17 698 844 479 479
q18 6569 5781 5926 5781
q19 1577 1116 679 679
q20 566 519 446 446
q21 2660 1996 1933 1933
q22 368 333 286 286
Total cold run time: 102215 ms
Total hot run time: 30500 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4574 4594 4518 4518
q2 269 333 250 250
q3 2359 2889 2381 2381
q4 1461 1934 1393 1393
q5 4650 4981 4650 4650
q6 239 186 142 142
q7 2024 1914 1784 1784
q8 2622 2462 2414 2414
q9 7639 7535 7828 7535
q10 2949 3014 2589 2589
q11 556 474 469 469
q12 961 749 600 600
q13 3884 4394 3392 3392
q14 290 307 286 286
q15 846 791 792 791
q16 647 716 656 656
q17 1093 1313 1315 1313
q18 7552 7301 7236 7236
q19 827 824 819 819
q20 1937 2024 1844 1844
q21 4555 4145 4074 4074
q22 577 537 490 490
Total cold run time: 52511 ms
Total hot run time: 49626 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3849210178
<details>
<summary>TPC-H: <b>Total hot run time: 31900 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d495108da4901493962c299dbb6195f10b46e8a8, data reload: false
------ Round 1 ----------------------------------
q1 17658 5300 5067 5067
q2 2007 315 216 216
q3 10214 1304 732 732
q4 10203 808 318 318
q5 7515 2156 1867 1867
q6 196 183 152 152
q7 882 762 623 623
q8 9284 1572 1067 1067
q9 5351 4756 4840 4756
q10 6810 1951 1568 1568
q11 528 324 273 273
q12 340 377 222 222
q13 17778 4026 3230 3230
q14 238 234 227 227
q15 937 812 822 812
q16 682 673 617 617
q17 641 836 447 447
q18 6610 6641 6700 6641
q19 1118 1002 641 641
q20 400 347 238 238
q21 2652 2099 1916 1916
q22 346 308 270 270
Total cold run time: 102390 ms
Total hot run time: 31900 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5331 5268 5290 5268
q2 259 324 252 252
q3 2153 2700 2227 2227
q4 1296 1716 1355 1355
q5 4263 4233 4227 4227
q6 230 184 139 139
q7 2258 2043 1888 1888
q8 2644 2511 2455 2455
q9 7681 7532 7521 7521
q10 2800 3003 2614 2614
q11 550 462 435 435
q12 682 782 668 668
q13 3907 4390 3525 3525
q14 311 334 310 310
q15 876 833 814 814
q16 719 766 741 741
q17 1235 1474 1411 1411
q18 7886 8042 7765 7765
q19 881 861 886 861
q20 2131 2195 2008 2008
q21 4621 4223 4112 4112
q22 605 538 509 509
Total cold run time: 53319 ms
Total hot run time: 51105 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3850671363
<details>
<summary>ClickBench: <b>Total hot run time: 28.74 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 26026e371347a7ccae8f5dc2d04f63f340f73d72, data reload: false
query1 0.05 0.04 0.05
query2 0.10 0.04 0.04
query3 0.26 0.09 0.09
query4 1.61 0.12 0.11
query5 0.26 0.26 0.24
query6 1.17 0.67 0.68
query7 0.02 0.03 0.02
query8 0.05 0.03 0.04
query9 0.57 0.50 0.49
query10 0.55 0.54 0.54
query11 0.15 0.09 0.10
query12 0.15 0.11 0.11
query13 0.64 0.62 0.62
query14 1.07 1.05 1.04
query15 0.88 0.88 0.89
query16 0.40 0.39 0.39
query17 1.13 1.16 1.13
query18 0.23 0.21 0.21
query19 2.15 1.99 1.96
query20 0.02 0.01 0.01
query21 15.39 0.25 0.16
query22 5.10 0.06 0.06
query23 15.97 0.29 0.11
query24 1.52 0.64 1.03
query25 0.09 0.08 0.08
query26 0.15 0.14 0.14
query27 0.09 0.06 0.06
query28 4.75 1.14 0.96
query29 12.56 3.97 3.18
query30 0.28 0.14 0.13
query31 2.82 0.65 0.43
query32 3.24 0.59 0.51
query33 3.29 3.22 3.22
query34 16.41 5.37 4.70
query35 4.86 4.75 4.80
query36 0.65 0.51 0.50
query37 0.12 0.07 0.07
query38 0.07 0.04 0.04
query39 0.05 0.03 0.03
query40 0.20 0.16 0.16
query41 0.09 0.03 0.03
query42 0.05 0.03 0.03
query43 0.05 0.04 0.04
Total cold run time: 99.26 s
Total hot run time: 28.74 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3862274175
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3850741133
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3862956812
# FE Regression Coverage Report
Increment line coverage ` 50.00% (106/212)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60414_1e8a877cc6dbb43ae659d6c536e623e4b0f86721_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60414_1e8a877cc6dbb43ae659d6c536e623e4b0f86721_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3849033069
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3850535277
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3839241622
<details>
<summary>ClickBench: <b>Total hot run time: 28.39 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 34a6005d0fe25a8f476efba19b90447178ad758f, data reload: false
query1 0.05 0.04 0.04
query2 0.10 0.05 0.04
query3 0.26 0.09 0.08
query4 1.62 0.12 0.11
query5 0.28 0.25 0.25
query6 1.17 0.68 0.66
query7 0.03 0.02 0.02
query8 0.05 0.04 0.04
query9 0.57 0.49 0.50
query10 0.54 0.54 0.54
query11 0.14 0.09 0.10
query12 0.15 0.10 0.11
query13 0.64 0.61 0.61
query14 1.04 1.08 1.06
query15 0.87 0.86 0.86
query16 0.39 0.39 0.44
query17 1.09 1.13 1.15
query18 0.23 0.21 0.21
query19 2.11 2.01 2.05
query20 0.02 0.01 0.02
query21 15.40 0.26 0.15
query22 5.16 0.06 0.06
query23 15.97 0.28 0.10
query24 1.42 0.36 0.59
query25 0.09 0.08 0.06
query26 0.13 0.13 0.14
query27 0.07 0.07 0.06
query28 4.20 1.13 0.96
query29 12.58 3.91 3.13
query30 0.27 0.12 0.12
query31 2.81 0.64 0.40
query32 3.24 0.59 0.50
query33 3.23 3.29 3.21
query34 16.23 5.36 4.79
query35 4.77 4.76 4.81
query36 0.65 0.50 0.49
query37 0.10 0.07 0.07
query38 0.08 0.04 0.04
query39 0.04 0.03 0.03
query40 0.19 0.16 0.14
query41 0.09 0.03 0.04
query42 0.05 0.03 0.03
query43 0.05 0.04 0.03
Total cold run time: 98.17 s
Total hot run time: 28.39 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3856895306
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3849845892
# FE Regression Coverage Report
Increment line coverage ` 44.30% (66/149)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60414_d495108da4901493962c299dbb6195f10b46e8a8_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60414_d495108da4901493962c299dbb6195f10b46e8a8_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3857051230
<details>
<summary>ClickBench: <b>Total hot run time: 28.13 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4702df99f971bf62d568f153c94dcbc4111b3f1f, data reload: false
query1 0.05 0.04 0.04
query2 0.10 0.05 0.05
query3 0.26 0.09 0.08
query4 1.60 0.11 0.11
query5 0.26 0.25 0.24
query6 1.18 0.68 0.68
query7 0.03 0.03 0.03
query8 0.06 0.04 0.04
query9 0.56 0.51 0.50
query10 0.54 0.55 0.55
query11 0.15 0.09 0.09
query12 0.13 0.10 0.11
query13 0.63 0.61 0.61
query14 1.06 1.05 1.06
query15 0.87 0.87 0.86
query16 0.39 0.38 0.38
query17 1.17 1.15 1.10
query18 0.22 0.21 0.21
query19 2.12 1.94 2.06
query20 0.02 0.02 0.01
query21 15.39 0.27 0.15
query22 5.20 0.05 0.04
query23 15.95 0.29 0.11
query24 1.42 0.27 0.18
query25 0.12 0.05 0.05
query26 0.14 0.13 0.14
query27 0.10 0.05 0.06
query28 3.18 1.16 0.97
query29 12.56 3.93 3.20
query30 0.28 0.13 0.11
query31 2.81 0.62 0.41
query32 3.24 0.59 0.49
query33 3.22 3.28 3.19
query34 16.83 5.37 4.70
query35 4.73 4.83 4.81
query36 0.65 0.50 0.49
query37 0.11 0.07 0.06
query38 0.07 0.04 0.04
query39 0.05 0.03 0.03
query40 0.20 0.16 0.16
query41 0.09 0.03 0.03
query42 0.04 0.02 0.03
query43 0.04 0.04 0.04
Total cold run time: 97.82 s
Total hot run time: 28.13 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3839015394
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3850631540
<details>
<summary>TPC-H: <b>Total hot run time: 32219 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26026e371347a7ccae8f5dc2d04f63f340f73d72, data reload: false
------ Round 1 ----------------------------------
q1 17611 5165 5072 5072
q2 2008 300 227 227
q3 10216 1315 764 764
q4 10235 918 314 314
q5 7539 2122 1935 1935
q6 201 181 146 146
q7 887 747 598 598
q8 9280 1372 1112 1112
q9 5276 4974 4861 4861
q10 6844 1948 1569 1569
q11 536 276 283 276
q12 329 375 222 222
q13 17781 4031 3201 3201
q14 239 247 213 213
q15 898 827 823 823
q16 688 678 621 621
q17 635 749 515 515
q18 6734 6474 7706 6474
q19 1619 1032 680 680
q20 412 385 236 236
q21 2907 2149 2078 2078
q22 382 347 282 282
Total cold run time: 103257 ms
Total hot run time: 32219 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5610 5564 5678 5564
q2 309 350 258 258
q3 2377 2820 2421 2421
q4 1457 1914 1471 1471
q5 4679 4611 4612 4611
q6 228 185 143 143
q7 2041 1862 1783 1783
q8 2748 2403 2379 2379
q9 7547 7320 7454 7320
q10 2751 3071 2638 2638
q11 552 471 471 471
q12 632 691 605 605
q13 3574 4005 3255 3255
q14 276 292 265 265
q15 833 812 788 788
q16 642 674 634 634
q17 1068 1219 1267 1219
q18 7573 7291 7163 7163
q19 810 791 795 791
q20 1947 2033 1899 1899
q21 4486 4302 4017 4017
q22 566 546 512 512
Total cold run time: 52706 ms
Total hot run time: 50207 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3862470838
<details>
<summary>ClickBench: <b>Total hot run time: 28.53 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1e8a877cc6dbb43ae659d6c536e623e4b0f86721, data reload: false
query1 0.05 0.05 0.05
query2 0.09 0.05 0.05
query3 0.26 0.09 0.08
query4 1.60 0.11 0.12
query5 0.26 0.25 0.25
query6 1.16 0.68 0.68
query7 0.03 0.02 0.03
query8 0.06 0.04 0.04
query9 0.58 0.49 0.49
query10 0.54 0.54 0.53
query11 0.14 0.09 0.10
query12 0.14 0.11 0.10
query13 0.62 0.60 0.62
query14 1.05 1.05 1.06
query15 0.89 0.88 0.87
query16 0.41 0.39 0.39
query17 1.17 1.10 1.15
query18 0.22 0.21 0.22
query19 2.07 2.02 2.03
query20 0.02 0.01 0.01
query21 15.44 0.25 0.14
query22 5.13 0.06 0.05
query23 16.16 0.28 0.12
query24 1.56 0.27 0.74
query25 0.07 0.06 0.06
query26 0.15 0.13 0.12
query27 0.06 0.08 0.06
query28 4.51 1.16 0.98
query29 12.56 3.96 3.18
query30 0.28 0.13 0.12
query31 2.82 0.65 0.42
query32 3.24 0.58 0.49
query33 3.28 3.32 3.32
query34 16.68 5.38 4.79
query35 4.74 4.81 4.78
query36 0.65 0.49 0.50
query37 0.11 0.07 0.07
query38 0.07 0.05 0.04
query39 0.05 0.03 0.03
query40 0.21 0.16 0.16
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 99.3 s
Total hot run time: 28.53 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3850722877
# FE UT Coverage Report
Increment line coverage ` 59.91% (127/212)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60414_26026e371347a7ccae8f5dc2d04f63f340f73d72/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60414_26026e371347a7ccae8f5dc2d04f63f340f73d72/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3889101888
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #60414:
URL: https://github.com/apache/doris/pull/60414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3857004240
<details>
<summary>TPC-H: <b>Total hot run time: 30957 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4702df99f971bf62d568f153c94dcbc4111b3f1f, data reload: false
------ Round 1 ----------------------------------
q1 17653 4480 4291 4291
q2 2022 337 257 257
q3 10124 1265 699 699
q4 10215 867 312 312
q5 7539 2138 1878 1878
q6 187 177 143 143
q7 841 713 583 583
q8 9261 1333 1072 1072
q9 5234 4889 4873 4873
q10 6818 1959 1569 1569
q11 497 293 283 283
q12 333 367 223 223
q13 17802 4024 3220 3220
q14 249 232 218 218
q15 886 809 803 803
q16 678 688 615 615
q17 643 817 489 489
q18 6658 6478 6338 6338
q19 1247 1013 633 633
q20 391 348 246 246
q21 2524 2047 1931 1931
q22 355 307 281 281
Total cold run time: 102157 ms
Total hot run time: 30957 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4340 4360 4318 4318
q2 258 341 258 258
q3 2112 2682 2231 2231
q4 1308 1739 1342 1342
q5 4363 4210 4316 4210
q6 224 190 142 142
q7 1854 1793 1746 1746
q8 2802 2530 2449 2449
q9 7664 7589 7555 7555
q10 2786 3009 2568 2568
q11 542 495 465 465
q12 684 732 603 603
q13 3787 4384 3831 3831
q14 299 325 287 287
q15 924 821 806 806
q16 660 728 688 688
q17 1180 1329 1438 1329
q18 8301 7885 7853 7853
q19 866 862 843 843
q20 2050 2171 2098 2098
q21 4610 4218 4004 4004
q22 557 560 495 495
Total cold run time: 52171 ms
Total hot run time: 50121 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3838846942
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3857124255
# FE UT Coverage Report
Increment line coverage ` 59.91% (127/212)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60414_4702df99f971bf62d568f153c94dcbc4111b3f1f/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60414_4702df99f971bf62d568f153c94dcbc4111b3f1f/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3850741040
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3849278000
<details>
<summary>ClickBench: <b>Total hot run time: 28.68 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d495108da4901493962c299dbb6195f10b46e8a8, data reload: false
query1 0.06 0.05 0.05
query2 0.10 0.05 0.05
query3 0.26 0.09 0.09
query4 1.61 0.11 0.11
query5 0.27 0.25 0.26
query6 1.18 0.69 0.68
query7 0.03 0.03 0.03
query8 0.06 0.04 0.04
query9 0.57 0.52 0.50
query10 0.56 0.57 0.54
query11 0.14 0.11 0.09
query12 0.13 0.10 0.10
query13 0.63 0.62 0.61
query14 1.07 1.05 1.07
query15 0.89 0.87 0.88
query16 0.38 0.38 0.38
query17 1.16 1.09 1.15
query18 0.23 0.22 0.21
query19 1.98 2.03 1.98
query20 0.02 0.02 0.01
query21 15.44 0.24 0.14
query22 5.15 0.07 0.05
query23 15.90 0.28 0.10
query24 1.57 0.65 1.00
query25 0.08 0.06 0.07
query26 0.14 0.13 0.13
query27 0.11 0.06 0.05
query28 4.66 1.15 0.97
query29 12.56 3.98 3.18
query30 0.27 0.13 0.11
query31 2.83 0.63 0.42
query32 3.25 0.60 0.49
query33 3.32 3.25 3.20
query34 16.07 5.35 4.73
query35 4.82 4.80 4.84
query36 0.65 0.50 0.49
query37 0.12 0.07 0.06
query38 0.08 0.05 0.04
query39 0.05 0.03 0.03
query40 0.19 0.16 0.15
query41 0.08 0.04 0.03
query42 0.05 0.03 0.03
query43 0.05 0.04 0.04
Total cold run time: 98.77 s
Total hot run time: 28.68 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3839185134
<details>
<summary>TPC-H: <b>Total hot run time: 31856 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 34a6005d0fe25a8f476efba19b90447178ad758f, data reload: false
------ Round 1 ----------------------------------
q1 17648 5370 5053 5053
q2 2125 301 195 195
q3 10144 1321 741 741
q4 10205 818 313 313
q5 7501 2155 1916 1916
q6 199 184 148 148
q7 873 740 615 615
q8 9262 1403 1111 1111
q9 5206 4778 4888 4778
q10 6781 1948 1575 1575
q11 504 282 279 279
q12 338 379 223 223
q13 17804 4045 3250 3250
q14 243 241 218 218
q15 914 819 835 819
q16 687 677 624 624
q17 630 757 537 537
q18 6925 6444 6575 6444
q19 1852 992 614 614
q20 405 346 241 241
q21 2639 2016 1893 1893
q22 358 312 269 269
Total cold run time: 103243 ms
Total hot run time: 31856 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5408 5308 5288 5288
q2 253 333 266 266
q3 2151 2715 2288 2288
q4 1333 1728 1334 1334
q5 4264 4120 4200 4120
q6 228 181 142 142
q7 2299 1971 1878 1878
q8 2634 2484 2459 2459
q9 7557 7501 7397 7397
q10 2858 3043 2524 2524
q11 540 469 464 464
q12 679 760 624 624
q13 3958 4436 3508 3508
q14 310 313 313 313
q15 941 861 870 861
q16 677 775 709 709
q17 1145 1377 1425 1377
q18 7925 7769 7879 7769
q19 884 865 921 865
q20 2091 2139 1918 1918
q21 4487 4201 4080 4080
q22 589 518 542 518
Total cold run time: 53211 ms
Total hot run time: 50702 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #60414:
URL: https://github.com/apache/doris/pull/60414#discussion_r2751942814
##########
fe/fe-core/src/test/java/org/apache/doris/common/util/S3UtilTest.java:
##########
@@ -248,5 +251,108 @@ public void testExtendGlobs() {
String result = S3Util.extendGlobs(input);
Assert.assertEquals(expected, result);
}
+
+ // Tests for isDeterministicPattern
+
+ @Test
+ public void testIsDeterministicPattern_simpleFile() {
+ // Simple file path without any patterns
+ Assert.assertTrue(S3Util.isDeterministicPattern("path/to/file.csv"));
+ }
+
+ @Test
+ public void testIsDeterministicPattern_withBraces() {
+ // Path with brace pattern (deterministic - can be expanded)
+ Assert.assertTrue(S3Util.isDeterministicPattern("path/to/file{1,2,3}.csv"));
+ Assert.assertTrue(S3Util.isDeterministicPattern("path/to/file{1..3}.csv"));
+ }
+
+ @Test
+ public void testIsDeterministicPattern_withAsterisk() {
+ // Path with asterisk wildcard (not deterministic)
+ Assert.assertFalse(S3Util.isDeterministicPattern("path/to/*.csv"));
+ Assert.assertFalse(S3Util.isDeterministicPattern("path/*/file.csv"));
+ }
+
+ @Test
+ public void testIsDeterministicPattern_withQuestionMark() {
+ // Path with question mark wildcard (not deterministic)
+ Assert.assertFalse(S3Util.isDeterministicPattern("path/to/file?.csv"));
+ }
+
+ @Test
+ public void testIsDeterministicPattern_withBrackets() {
+ // Path with bracket pattern (not deterministic)
+ Assert.assertFalse(S3Util.isDeterministicPattern("path/to/file[0-9].csv"));
+ Assert.assertFalse(S3Util.isDeterministicPattern("path/to/file[abc].csv"));
+ }
+
+ @Test
+ public void testIsDeterministicPattern_withEscape() {
+ // Path with escape character (not deterministic - complex pattern)
+ Assert.assertFalse(S3Util.isDeterministicPattern("path/to/file\\*.csv"));
+ }
+
+ @Test
+ public void testIsDeterministicPattern_mixed() {
+ // Path with both braces and wildcards
+ Assert.assertFalse(S3Util.isDeterministicPattern("path/to/file{1,2}/*.csv"));
+ }
+
+ // Tests for expandBracePatterns
+
+ @Test
+ public void testExpandBracePatterns_noBraces() {
+ // No braces - returns single path
+ List<String> result = S3Util.expandBracePatterns("path/to/file.csv");
+ Assert.assertEquals(Arrays.asList("path/to/file.csv"), result);
+ }
+
+ @Test
+ public void testExpandBracePatterns_simpleBrace() {
+ // Simple brace expansion
+ List<String> result = S3Util.expandBracePatterns("file{1,2,3}.csv");
+ Assert.assertEquals(Arrays.asList("file1.csv", "file2.csv", "file3.csv"), result);
+ }
+
+ @Test
+ public void testExpandBracePatterns_multipleBraces() {
+ // Multiple brace expansions
+ List<String> result = S3Util.expandBracePatterns("dir{a,b}/file{1,2}.csv");
+ Assert.assertEquals(Arrays.asList(
+ "dira/file1.csv", "dira/file2.csv",
+ "dirb/file1.csv", "dirb/file2.csv"), result);
+ }
+
+ @Test
+ public void testExpandBracePatterns_emptyBrace() {
+ // Empty brace content
+ List<String> result = S3Util.expandBracePatterns("file{}.csv");
+ Assert.assertEquals(Arrays.asList("file.csv"), result);
+ }
+
+ @Test
+ public void testExpandBracePatterns_singleValue() {
+ // Single value in brace
+ List<String> result = S3Util.expandBracePatterns("file{1}.csv");
+ Assert.assertEquals(Arrays.asList("file1.csv"), result);
+ }
+
+ @Test
+ public void testExpandBracePatterns_withPath() {
+ // Full path with braces
+ List<String> result = S3Util.expandBracePatterns("data/year{2023,2024}/month{01,02}/file.csv");
+ Assert.assertEquals(8, result.size());
Review Comment:
The expected result size is incorrect. The pattern `"data/year{2023,2024}/month{01,02}/file.csv"` should expand to 4 paths (2 years × 2 months), not 8. The correct expansion should produce:
1. `data/year2023/month01/file.csv`
2. `data/year2023/month02/file.csv`
3. `data/year2024/month01/file.csv`
4. `data/year2024/month02/file.csv`
Change the expected size from 8 to 4.
```suggestion
Assert.assertEquals(4, result.size());
```
##########
fe/fe-core/src/main/java/org/apache/doris/fs/obj/S3ObjStorage.java:
##########
@@ -578,6 +579,23 @@ private GlobListResult globListInternal(String remotePath, List<RemoteFile> resu
}
bucket = uri.getBucket();
+
+ // Optimization: For deterministic paths (no wildcards like *, ?, [...]),
+ // use HEAD requests instead of listing to avoid requiring ListBucket permission.
+ // This is useful when only GetObject permission is granted.
+ // Controlled by config: s3_skip_list_for_deterministic_path
+ String keyPattern = uri.getKey();
+ if (Config.s3_skip_list_for_deterministic_path
+ && S3Util.isDeterministicPattern(keyPattern)
Review Comment:
No limit on the number of expanded paths. A pattern like `file{{1..1000}}/dir{{1..1000}}/data.csv` would expand to 1 million paths and trigger 1 million HEAD requests, which could overwhelm the S3 service or cause performance issues. Consider adding a reasonable upper limit (e.g., 1000 or 10000 paths) and either throwing an error or falling back to the LIST-based approach when this limit is exceeded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](s3) Skip S3 listing for deterministic file paths using HEAD requests [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60414:
URL: https://github.com/apache/doris/pull/60414#issuecomment-3831979488
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org