You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cloud-fan (via GitHub)" <gi...@apache.org> on 2026/04/07 16:14:46 UTC

[PR] [SPARK-56338][INFRA] Support MAVEN_MIRROR_URL in SBT launcher bootstrap [spark]

cloud-fan opened a new pull request, #55238:
URL: https://github.com/apache/spark/pull/55238

   ### What changes were proposed in this pull request?
   
   This is a follow-up to #55168. The SBT launcher script (`build/sbt-launch-lib.bash`) did not respect `MAVEN_MIRROR_URL`. This patch adds support for it in two places:
   
   1. **Launcher JAR download**: `MAVEN_MIRROR_URL` is used as the default for `DEFAULT_ARTIFACT_REPOSITORY` when neither is explicitly set.
   2. **SBT boot phase**: When `MAVEN_MIRROR_URL` is set, a temporary SBT repositories config is generated and passed via `-Dsbt.repository.config` so SBT resolves itself and Scala through the mirror.
   
   ### Why are the changes needed?
   
   SPARK-56338 added `MAVEN_MIRROR_URL` support for Maven (`pom.xml`) and SBT project builds (`SparkBuild.scala`), but the SBT launcher script was not covered. In environments where default Maven repositories are unreachable, the SBT launcher JAR download and boot phase still fail without manual configuration (e.g. `~/.sbt/repositories`).
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Manually tested by setting `MAVEN_MIRROR_URL` and building Spark with SBT from scratch (launcher JAR removed, no `~/.sbt/repositories`).
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56338][INFRA][FOLLOWUP] Support MAVEN_MIRROR_URL in SBT launcher bootstrap [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #55238:
URL: https://github.com/apache/spark/pull/55238#issuecomment-4210765687

   the docker test failure is unrelated, thanks for review, merging to master!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56338][INFRA][FOLLOWUP] Support MAVEN_MIRROR_URL in SBT launcher bootstrap [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #55238:
URL: https://github.com/apache/spark/pull/55238#discussion_r3049250544


##########
build/sbt-launch-lib.bash:
##########
@@ -182,6 +183,18 @@ run() {
   set -- "${residual_args[@]}"
   argumentCount=$#
 
+  # If MAVEN_MIRROR_URL is set, generate a repositories config so the SBT launcher
+  # resolves SBT and Scala through the mirror during the boot phase.
+  if [[ -n "$MAVEN_MIRROR_URL" ]]; then
+    local sbt_repo_config="$(mktemp /tmp/sbt-repositories.XXXXXX)"

Review Comment:
   The SBT bootstrap phase only downloads SBT itself and Scala — a single mirror should be sufficient for that. This is just to make `MAVEN_MIRROR_URL` work end-to-end consistently with the existing support in `pom.xml` and `SparkBuild.scala`.
   
   For more complex multi-repo setups, users can still configure `~/.sbt/repositories` or `-Dsbt.repository.config` directly.



##########
build/sbt-launch-lib.bash:
##########
@@ -182,6 +183,18 @@ run() {
   set -- "${residual_args[@]}"
   argumentCount=$#
 
+  # If MAVEN_MIRROR_URL is set, generate a repositories config so the SBT launcher
+  # resolves SBT and Scala through the mirror during the boot phase.
+  if [[ -n "$MAVEN_MIRROR_URL" ]]; then
+    local sbt_repo_config="$(mktemp /tmp/sbt-repositories.XXXXXX)"
+    cat > "$sbt_repo_config" <<EOF
+[repositories]
+  local
+  maven-mirror: ${MAVEN_MIRROR_URL}
+EOF
+    addJava "-Dsbt.repository.config=$sbt_repo_config"

Review Comment:
   Good points. I'll add a guard to skip if `-Dsbt.repository.config` is already in `SBT_OPTS`, and add a trap to clean up the temp file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56338][INFRA][FOLLOWUP] Support MAVEN_MIRROR_URL in SBT launcher bootstrap [spark]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #55238:
URL: https://github.com/apache/spark/pull/55238#discussion_r3049028848


##########
build/sbt-launch-lib.bash:
##########
@@ -182,6 +183,18 @@ run() {
   set -- "${residual_args[@]}"
   argumentCount=$#
 
+  # If MAVEN_MIRROR_URL is set, generate a repositories config so the SBT launcher
+  # resolves SBT and Scala through the mirror during the boot phase.
+  if [[ -n "$MAVEN_MIRROR_URL" ]]; then
+    local sbt_repo_config="$(mktemp /tmp/sbt-repositories.XXXXXX)"

Review Comment:
   it's a little bit hacky, users may want to configure more than one repo sometimes
   
   in Apache Celeborn, we allow users to have a `build/sbt-config/repositories-local` (ignored by Git), and provide several common templates
   
   https://github.com/apache/celeborn/blob/main/build/sbt-config/



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56338][INFRA][FOLLOWUP] Support MAVEN_MIRROR_URL in SBT launcher bootstrap [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #55238:
URL: https://github.com/apache/spark/pull/55238#issuecomment-4200528242

   cc @gaogaotiantian @zhengruifeng 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56338][INFRA][FOLLOWUP] Support MAVEN_MIRROR_URL in SBT launcher bootstrap [spark]

Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on code in PR #55238:
URL: https://github.com/apache/spark/pull/55238#discussion_r3046966899


##########
build/sbt-launch-lib.bash:
##########
@@ -182,6 +183,18 @@ run() {
   set -- "${residual_args[@]}"
   argumentCount=$#
 
+  # If MAVEN_MIRROR_URL is set, generate a repositories config so the SBT launcher
+  # resolves SBT and Scala through the mirror during the boot phase.
+  if [[ -n "$MAVEN_MIRROR_URL" ]]; then
+    local sbt_repo_config="$(mktemp /tmp/sbt-repositories.XXXXXX)"
+    cat > "$sbt_repo_config" <<EOF
+[repositories]
+  local
+  maven-mirror: ${MAVEN_MIRROR_URL}
+EOF
+    addJava "-Dsbt.repository.config=$sbt_repo_config"

Review Comment:
   If `-Dsbt.repository.config` is already set by the users, what would happen? We still want that to work right? Also maybe we should cleanup the temp dir after the script? Even if it's in `/tmp`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56338][INFRA][FOLLOWUP] Support MAVEN_MIRROR_URL in SBT launcher bootstrap [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #55238: [SPARK-56338][INFRA][FOLLOWUP] Support MAVEN_MIRROR_URL in SBT launcher bootstrap
URL: https://github.com/apache/spark/pull/55238


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org