You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xylaaaaa (via GitHub)" <gi...@apache.org> on 2026/03/23 09:52:27 UTC
[PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
xylaaaaa opened a new pull request, #61632:
URL: https://github.com/apache/doris/pull/61632
## Summary
- make `run-thirdparties-docker.sh` resolve `juicefs-helpers.sh` from multiple candidate roots instead of assuming a fixed repo layout
- fall back to built-in minimal JuiceFS helper functions when the helper file is unavailable in CI packaging layouts
- keep the change scoped to the thirdparties startup script without touching external regression defaults
## Test Plan
- bash -n docker/thirdparties/run-thirdparties-docker.sh
- git diff --check
- simulate a CI layout that only contains `docker/thirdparties` and verify the script no longer fails on missing `thirdparty/juicefs-helpers.sh`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61632:
URL: https://github.com/apache/doris/pull/61632#issuecomment-4115253446
<details>
<summary>TPC-DS: <b>Total hot run time: 168803 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f960716297e0fc6eca3ed852e0a9e2cfcd93d5d9, data reload: false
query5 4330 633 482 482
query6 326 227 207 207
query7 4230 471 270 270
query8 335 239 233 233
query9 8694 2683 2705 2683
query10 491 373 323 323
query11 6988 5079 4896 4896
query12 188 133 127 127
query13 1271 453 350 350
query14 5747 3729 3412 3412
query14_1 2839 2831 2841 2831
query15 203 194 177 177
query16 1007 485 448 448
query17 1111 747 628 628
query18 2444 452 355 355
query19 222 218 188 188
query20 141 129 126 126
query21 219 135 111 111
query22 13264 14050 14801 14050
query23 16584 16117 15987 15987
query23_1 15962 16256 15943 15943
query24 7162 1633 1213 1213
query24_1 1221 1218 1226 1218
query25 552 460 397 397
query26 1235 263 151 151
query27 2778 476 297 297
query28 4457 1873 1802 1802
query29 850 597 477 477
query30 291 227 188 188
query31 1004 940 861 861
query32 84 68 68 68
query33 513 342 274 274
query34 878 880 497 497
query35 643 668 590 590
query36 1065 1133 998 998
query37 144 94 80 80
query38 2948 2941 2931 2931
query39 859 845 812 812
query39_1 801 795 795 795
query40 231 155 131 131
query41 61 60 57 57
query42 258 252 250 250
query43 252 246 223 223
query44
query45 194 187 182 182
query46 872 977 608 608
query47 2128 2161 2036 2036
query48 333 321 223 223
query49 625 463 368 368
query50 719 280 209 209
query51 4113 4098 3993 3993
query52 260 263 257 257
query53 290 335 281 281
query54 332 270 263 263
query55 91 84 85 84
query56 328 316 296 296
query57 1902 1720 1661 1661
query58 286 271 273 271
query59 2783 2944 2761 2761
query60 344 339 326 326
query61 155 150 152 150
query62 619 572 536 536
query63 302 283 270 270
query64 5077 1270 1001 1001
query65
query66 1465 450 363 363
query67 24219 24190 24104 24104
query68
query69 403 313 279 279
query70 968 975 936 936
query71 332 315 281 281
query72 2877 2747 2624 2624
query73 545 547 317 317
query74 9630 9580 9396 9396
query75 2931 2771 2470 2470
query76 2308 1066 682 682
query77 405 420 327 327
query78 10949 11119 10464 10464
query79 1142 787 568 568
query80 1375 650 568 568
query81 544 269 230 230
query82 1031 182 121 121
query83 331 264 244 244
query84 254 121 93 93
query85 912 483 438 438
query86 413 308 294 294
query87 3135 3104 3064 3064
query88 3599 2652 2655 2652
query89 426 363 343 343
query90 2046 187 166 166
query91 165 156 141 141
query92 77 72 73 72
query93 938 846 483 483
query94 651 322 279 279
query95 599 333 324 324
query96 644 515 231 231
query97 2444 2526 2392 2392
query98 229 219 215 215
query99 999 987 941 941
Total cold run time: 249670 ms
Total hot run time: 168803 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61632:
URL: https://github.com/apache/doris/pull/61632#issuecomment-4114102121
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen merged PR #61632:
URL: https://github.com/apache/doris/pull/61632
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #61632:
URL: https://github.com/apache/doris/pull/61632#issuecomment-4115123924
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61632:
URL: https://github.com/apache/doris/pull/61632#issuecomment-4109337936
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #61632:
URL: https://github.com/apache/doris/pull/61632#issuecomment-4115061490
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61632:
URL: https://github.com/apache/doris/pull/61632#issuecomment-4115223728
<details>
<summary>TPC-H: <b>Total hot run time: 26596 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f960716297e0fc6eca3ed852e0a9e2cfcd93d5d9, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17621 4520 4277 4277
q2 q3 10651 752 510 510
q4 4665 342 248 248
q5 7569 1224 1032 1032
q6 176 173 146 146
q7 778 830 665 665
q8 9484 1423 1313 1313
q9 4911 4731 4707 4707
q10 6326 1910 1663 1663
q11 467 254 238 238
q12 772 579 459 459
q13 18062 2702 1933 1933
q14 220 230 219 219
q15 q16 750 728 677 677
q17 735 826 499 499
q18 5975 5373 5217 5217
q19 1205 994 633 633
q20 534 499 383 383
q21 4534 1838 1452 1452
q22 336 470 325 325
Total cold run time: 95771 ms
Total hot run time: 26596 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4785 4572 4608 4572
q2 q3 3864 4359 3802 3802
q4 949 1223 825 825
q5 4092 4424 4352 4352
q6 192 187 145 145
q7 1739 1666 1578 1578
q8 2533 2731 2569 2569
q9 7555 7423 7265 7265
q10 3739 3960 3621 3621
q11 523 446 450 446
q12 527 620 475 475
q13 2664 2813 1995 1995
q14 300 291 282 282
q15 q16 744 762 719 719
q17 1183 1389 1387 1387
q18 7402 6849 6499 6499
q19 948 899 942 899
q20 2052 2181 2223 2181
q21 4025 3539 3360 3360
q22 481 424 388 388
Total cold run time: 50297 ms
Total hot run time: 47360 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
Posted by "xylaaaaa (via GitHub)" <gi...@apache.org>.
xylaaaaa commented on PR #61632:
URL: https://github.com/apache/doris/pull/61632#issuecomment-4109497960
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61632:
URL: https://github.com/apache/doris/pull/61632#issuecomment-4115557999
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #61632:
URL: https://github.com/apache/doris/pull/61632#discussion_r2973974644
##########
docker/thirdparties/run-thirdparties-docker.sh:
##########
@@ -231,8 +231,91 @@ reserve_ports() {
}
JFS_META_FORMATTED=0
-DORIS_ROOT="$(cd "${ROOT}/../.." &>/dev/null && pwd)"
-. "${DORIS_ROOT}/thirdparty/juicefs-helpers.sh"
+DORIS_ROOT="${DORIS_ROOT:-$(cd "${ROOT}/../.." &>/dev/null && pwd)}"
+JUICEFS_DEFAULT_VERSION="${JUICEFS_DEFAULT_VERSION:-1.3.1}"
+JUICEFS_HADOOP_MAVEN_REPO="${JUICEFS_HADOOP_MAVEN_REPO:-https://repo1.maven.org/maven2/io/juicefs/juicefs-hadoop}"
+
+load_juicefs_helpers() {
+ local candidate=""
+ local -a helper_candidates=(
+ "${DORIS_ROOT}/thirdparty/juicefs-helpers.sh"
+ "${ROOT}/../../thirdparty/juicefs-helpers.sh"
+ "$(pwd)/thirdparty/juicefs-helpers.sh"
+ )
+
+ for candidate in "${helper_candidates[@]}"; do
+ if [[ -f "${candidate}" ]]; then
+ . "${candidate}"
+ return 0
Review Comment:
The helper existence check uses [[ -f "${candidate}" ]] before sourcing. If the file exists but is not readable, the subsequent '.' will fail and (with `set -e`) abort the script instead of falling back to the built-in helpers. Consider checking readability (e.g., -r) and/or handling a failed source by continuing to the next candidate / falling back.
```suggestion
if [[ -r "${candidate}" ]]; then
if . "${candidate}"; then
return 0
fi
```
##########
docker/thirdparties/run-thirdparties-docker.sh:
##########
@@ -231,8 +231,91 @@ reserve_ports() {
}
JFS_META_FORMATTED=0
-DORIS_ROOT="$(cd "${ROOT}/../.." &>/dev/null && pwd)"
-. "${DORIS_ROOT}/thirdparty/juicefs-helpers.sh"
+DORIS_ROOT="${DORIS_ROOT:-$(cd "${ROOT}/../.." &>/dev/null && pwd)}"
+JUICEFS_DEFAULT_VERSION="${JUICEFS_DEFAULT_VERSION:-1.3.1}"
+JUICEFS_HADOOP_MAVEN_REPO="${JUICEFS_HADOOP_MAVEN_REPO:-https://repo1.maven.org/maven2/io/juicefs/juicefs-hadoop}"
+
+load_juicefs_helpers() {
+ local candidate=""
+ local -a helper_candidates=(
+ "${DORIS_ROOT}/thirdparty/juicefs-helpers.sh"
+ "${ROOT}/../../thirdparty/juicefs-helpers.sh"
+ "$(pwd)/thirdparty/juicefs-helpers.sh"
+ )
+
+ for candidate in "${helper_candidates[@]}"; do
+ if [[ -f "${candidate}" ]]; then
+ . "${candidate}"
+ return 0
Review Comment:
The helper lookup includes "$(pwd)/thirdparty/juicefs-helpers.sh" as a source candidate. Sourcing from the current working directory can execute unintended/untrusted code if the script is run from a directory containing a spoofed thirdparty/juicefs-helpers.sh. Prefer restricting candidates to paths derived from ROOT/DORIS_ROOT (or require an explicit opt-in env var for CWD lookup) to avoid this execution risk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](docker) make JuiceFS helper lookup robust in thirdparties [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61632:
URL: https://github.com/apache/doris/pull/61632#issuecomment-4114101936
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org