You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "afterincomparableyum (via GitHub)" <gi...@apache.org> on 2026/10/17 03:58:42 UTC
[PR] [CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven Central failures [celeborn]
afterincomparableyum opened a new pull request, #3877:
URL: https://github.com/apache/celeborn/pull/3877
build/sbt fetched sbt-launch.jar from repo1.maven.org with a single `curl --silent` call and no retry. Any brief failure from Maven Central (DNS, connection reset, 403/429 throttling) failed the whole CI job before a single test ran, and the log never said why. The repo's Actions cache sits at its 10GB limit, so the cached launcher is usually evicted and every SBT matrix job downloads it at the same time.
Dependency resolution had the same weak point. build/sbt-config/repositories lists Google's Maven Central mirror ahead of repo1, but at the old /repos/central/data/ path, which now returns 404. Every artifact was really fetched from repo1 alone, so a repo1 connection reset failed the build (e.g. "could not retrieve sbt 1.9.4" when the launcher couldn't download scala-reflect).
- Retry the launcher download with curl/wget, and add --retry-all-errors when curl supports it (7.71+), because plain --retry skips DNS, connection and 403 errors.
- Print the download error instead of hiding it.
- If repo1 still fails, fetch the launcher from Google's Maven Central mirror. Skip that fallback when DEFAULT_ARTIFACT_REPOSITORY is set, so builds limited to an internal repo stay that way.
- Point gcs-maven-central-mirror at the live /maven2/ path, as repositories-asia.template already does, so dependencies have a working mirror ahead of repo1.
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR resolve a correctness bug?
<!-- Check if yes. The `correctness` label will be added/removed automatically. -->
- [ ] Yes
### Does this PR introduce _any_ user-facing change?
<!-- Check if yes. -->
- [ ] Yes
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven Central failures [celeborn]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 closed pull request #3877: [CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven Central failures
URL: https://github.com/apache/celeborn/pull/3877
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven Central failures [celeborn]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on code in PR #3877:
URL: https://github.com/apache/celeborn/pull/3877#discussion_r4311079873
##########
build/sbt-launch-lib.bash:
##########
@@ -73,16 +79,25 @@ acquire_sbt_jar () {
# Download
printf "Attempting to fetch sbt\n"
JAR_DL="${JAR}.part"
- if [ $(command -v curl) ]; then
- curl --fail --location --silent ${URL1} > "${JAR_DL}" &&\
- mv "${JAR_DL}" "${JAR}"
- elif [ $(command -v wget) ]; then
- wget --quiet ${URL1} -O "${JAR_DL}" &&\
- mv "${JAR_DL}" "${JAR}"
- else
- printf "You do not have curl or wget installed, please install sbt manually from https://www.scala-sbt.org/\n"
- exit -1
+ CURL_RETRY_OPTS="--retry 3 --retry-delay 5"
+ # --retry alone skips DNS, connection and 403 errors; --retry-all-errors needs curl 7.71+.
+ if [ $(command -v curl) ] && curl --help all 2>/dev/null | grep -q -- "--retry-all-errors"; then
+ CURL_RETRY_OPTS="${CURL_RETRY_OPTS} --retry-all-errors"
fi
+ for URL in ${URLS}; do
+ if [ $(command -v curl) ]; then
+ curl --fail --location --silent --show-error ${CURL_RETRY_OPTS} ${URL} > "${JAR_DL}" &&\
Review Comment:
`--retry-all-errors` can retry after a transfer has written a partial JAR. With `> "${JAR_DL}"`, the shell keeps that partial output across curl's retries, so a later successful attempt can append to it. Curl may then exit successfully and the following `mv` installs a corrupt JAR, bypassing the mirror fallback. The [curl manual](https://curl.se/docs/manpage.html#--retry-all-errors) explicitly warns against redirected output here. Could you use `--output "${JAR_DL}"` instead so curl can reset the file between attempts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven Central failures [celeborn]
Posted by "afterincomparableyum (via GitHub)" <gi...@apache.org>.
afterincomparableyum commented on code in PR #3877:
URL: https://github.com/apache/celeborn/pull/3877#discussion_r4311190005
##########
build/sbt-launch-lib.bash:
##########
@@ -73,16 +79,25 @@ acquire_sbt_jar () {
# Download
printf "Attempting to fetch sbt\n"
JAR_DL="${JAR}.part"
- if [ $(command -v curl) ]; then
- curl --fail --location --silent ${URL1} > "${JAR_DL}" &&\
- mv "${JAR_DL}" "${JAR}"
- elif [ $(command -v wget) ]; then
- wget --quiet ${URL1} -O "${JAR_DL}" &&\
- mv "${JAR_DL}" "${JAR}"
- else
- printf "You do not have curl or wget installed, please install sbt manually from https://www.scala-sbt.org/\n"
- exit -1
+ CURL_RETRY_OPTS="--retry 3 --retry-delay 5"
+ # --retry alone skips DNS, connection and 403 errors; --retry-all-errors needs curl 7.71+.
+ if [ $(command -v curl) ] && curl --help all 2>/dev/null | grep -q -- "--retry-all-errors"; then
+ CURL_RETRY_OPTS="${CURL_RETRY_OPTS} --retry-all-errors"
fi
+ for URL in ${URLS}; do
+ if [ $(command -v curl) ]; then
+ curl --fail --location --silent --show-error ${CURL_RETRY_OPTS} ${URL} > "${JAR_DL}" &&\
Review Comment:
thank you, I have applied the change @zaynt4606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven Central failures [celeborn]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on PR #3877:
URL: https://github.com/apache/celeborn/pull/3877#issuecomment-6263870219
**Merge Summary:**
- merged into main https://github.com/apache/celeborn/commit/eca67458
*Posted by `merge_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven Central failures [celeborn]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on PR #3877:
URL: https://github.com/apache/celeborn/pull/3877#issuecomment-6263892198
Thanks @afterincomparableyum for the contribution!
Merged to main(v1.0.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org