You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/25 19:18:38 UTC

[GitHub] [spark] ankits opened a new pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

ankits opened a new pull request #29874:
URL: https://github.com/apache/spark/pull/29874


   …nal repos only
   
   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714784073






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714842733


   **[Test build #130180 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130180/testReport)** for PR 29874 at commit [`cfb86c3`](https://github.com/apache/spark/commit/cfb86c3608481b8767ad6d104ce3a2884388c13d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714038764






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714659678


   **[Test build #130171 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130171/testReport)** for PR 29874 at commit [`b58631c`](https://github.com/apache/spark/commit/b58631c9414b74444fb20921cee56fa02bf3cae7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-706290156


   **[Test build #129590 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129590/testReport)** for PR 29874 at commit [`8da6fd5`](https://github.com/apache/spark/commit/8da6fd50dc3ef4c36e470b74b963476f064a43f2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714792987


   **[Test build #130180 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130180/testReport)** for PR 29874 at commit [`cfb86c3`](https://github.com/apache/spark/commit/cfb86c3608481b8767ad6d104ce3a2884388c13d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702977113


   **[Test build #129369 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129369/testReport)** for PR 29874 at commit [`1b0ba28`](https://github.com/apache/spark/commit/1b0ba28af9e3c7b80ebc095bea8b78b70c5b5c4a).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dbtsai commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699229945


   okay to test.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714765270






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714833305






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r505891657



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -61,7 +61,8 @@ private[hive] object IsolatedClientLoader extends Logging {
     val files = if (resolvedVersions.contains((resolvedVersion, hadoopVersion))) {
       resolvedVersions((resolvedVersion, hadoopVersion))
     } else {
-      val remoteRepos = sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)
+      val remoteRepos = sys.env.getOrElse(
+        "DEFAULT_ARTIFACT_REPOSITORY", sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))

Review comment:
       I was looking at something similar today and realized that what I said doesn't quite make sense, since you can't really externally configure `spark.sql.maven.additionalRemoteRepositories` when running the tests via Maven.
   
   I'm still not sure if configuring the default value of a config based on an environment variable makes sense, though. Maybe it would be better to update the tests somewhere (if that is indeed your concern). 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-704651444






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702985432






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-713890350


   **[Test build #130108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130108/testReport)** for PR 29874 at commit [`ff644eb`](https://github.com/apache/spark/commit/ff644eb0c291a4ede8ac0237ae31a3fd68a22a8e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709648538


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34465/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-712471695


   **[Test build #130023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130023/testReport)** for PR 29874 at commit [`11a9a67`](https://github.com/apache/spark/commit/11a9a67771c6e67153bb4ea9548a2921dd2af7ca).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r510485493



##########
File path: build/sbt-launch-lib.bash
##########
@@ -39,7 +39,12 @@ dlog () {
 
 acquire_sbt_jar () {
   SBT_VERSION=`awk -F "=" '/sbt\.version/ {print $2}' ./project/build.properties`
-  URL1=https://repo1.maven.org/maven2/org/scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch-${SBT_VERSION}.jar
+  #
+  # DEFAULT_ARTIFACT_REPOSITORY env variables can be used to only fetch

Review comment:
       `variables` -> `variable`.

##########
File path: build/sbt-launch-lib.bash
##########
@@ -39,7 +39,12 @@ dlog () {
 
 acquire_sbt_jar () {
   SBT_VERSION=`awk -F "=" '/sbt\.version/ {print $2}' ./project/build.properties`
-  URL1=https://repo1.maven.org/maven2/org/scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch-${SBT_VERSION}.jar
+  #

Review comment:
       Let's remove this empty comment.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] ankits commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
ankits commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714849993


   For `maven` we can setup  a `settings.xml` in `$M2_HOME` that points to internal artifact repository. Similar setup can be done for `sbt` where you can define the `-Dsbt.repository.config` to fetch artifacts from the internal artifact repository. 
   
   The main purpose of the PR was to address test execution and download of artifacts which wasn't supported with Dsbt.repository.config.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699618497


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-712563758


   **[Test build #130028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130028/testReport)** for PR 29874 at commit [`ff644eb`](https://github.com/apache/spark/commit/ff644eb0c291a4ede8ac0237ae31a3fd68a22a8e).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-712564310






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699618500


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129146/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702976554


   **[Test build #129369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129369/testReport)** for PR 29874 at commit [`1b0ba28`](https://github.com/apache/spark/commit/1b0ba28af9e3c7b80ebc095bea8b78b70c5b5c4a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714787050


   **[Test build #130178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130178/testReport)** for PR 29874 at commit [`91d0d4a`](https://github.com/apache/spark/commit/91d0d4a71a0abb389dd7b5ebe6db17947b00a260).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-706093252


   Hi, @ankits . The master branch is upgraded to SBT 1.3.13.
   Could you resolve the conflicts?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-712518103


   **[Test build #130028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130028/testReport)** for PR 29874 at commit [`ff644eb`](https://github.com/apache/spark/commit/ff644eb0c291a4ede8ac0237ae31a3fd68a22a8e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699604867


   **[Test build #129146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129146/testReport)** for PR 29874 at commit [`de4c4d7`](https://github.com/apache/spark/commit/de4c4d7d47fbea164edba148ddf2341bd7b065de).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-713904160


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34717/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714765270






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702985425


   **[Test build #129371 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129371/testReport)** for PR 29874 at commit [`12a06c0`](https://github.com/apache/spark/commit/12a06c042011ef8302ab2b61c935714c58e8453f).
    * This patch **fails Scala style tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714832623


   **[Test build #130177 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130177/testReport)** for PR 29874 at commit [`5a90eb7`](https://github.com/apache/spark/commit/5a90eb7974614edd961a946be3c93b28160a2136).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714836423






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714790013


   **[Test build #130179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130179/testReport)** for PR 29874 at commit [`f0ee8b8`](https://github.com/apache/spark/commit/f0ee8b83a84d89f2533e6684b740a623c1d893f2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-706093252


   Hi, @ankits . The master branch is upgraded to SBT 1.3.13.
   Could you resolve the conflicts?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-713915482






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] ankits commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
ankits commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r510290283



##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -1160,13 +1160,16 @@ private[spark] object SparkSubmitUtils {
     val br: IBiblioResolver = new IBiblioResolver
     br.setM2compatible(true)
     br.setUsepoms(true)
+    val defaultInternalRepo : Option[String] = sys.env.get("DEFAULT_ARTIFACT_REPOSITORY")
+    br.setRoot(defaultInternalRepo.getOrElse("https://repo1.maven.org/maven2/"))

Review comment:
       Yes. That is the default for `IBiblioResolver`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] ankits commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
ankits commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r499065894



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -61,7 +61,8 @@ private[hive] object IsolatedClientLoader extends Logging {
     val files = if (resolvedVersions.contains((resolvedVersion, hadoopVersion))) {
       resolvedVersions((resolvedVersion, hadoopVersion))
     } else {
-      val remoteRepos = sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)
+      val remoteRepos = sys.env.getOrElse(
+        "DEFAULT_ARTIFACT_REPOSITORY", sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))

Review comment:
       @xkrogen During my testing of your suggested changes, the test still tries to download the artifact from `SQLConf.ADDITIONAL_REMOTE_REPOSITORIES` which points to `https://maven-central.storage-download.googleapis.com/maven2/`. I still need the change in `SQLConf.scala` to overwrite the maven repo. 
   
   ```   val ADDITIONAL_REMOTE_REPOSITORIES =
       buildConf("spark.sql.maven.additionalRemoteRepositories")
         .doc("A comma-delimited string config of the optional additional remote Maven mirror " +
           "repositories. This is only used for downloading Hive jars in IsolatedClientLoader " +
           "if the default Maven Central repo is unreachable.")
         .version("3.0.0")
         .stringConf
         .createWithDefault(
           sys.env.getOrElse(
             "DEFAULT_ARTIFACT_REPOSITORY",
             "https://maven-central.storage-download.googleapis.com/maven2/"))
   ```
   
   
   Let me know your thoughts on this. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702985432


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-706344921


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-712513646


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130023/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699108878


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709631946


   **[Test build #129859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129859/testReport)** for PR 29874 at commit [`8da6fd5`](https://github.com/apache/spark/commit/8da6fd50dc3ef4c36e470b74b963476f064a43f2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-712513639


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714841081






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714835897


   **[Test build #130178 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130178/testReport)** for PR 29874 at commit [`91d0d4a`](https://github.com/apache/spark/commit/91d0d4a71a0abb389dd7b5ebe6db17947b00a260).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] ankits commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
ankits commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714659546


   I tested `master` internally just now and didn't need to update `pom.xml` or `project/SparkBuild.scala`.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702977127






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699604684


   ok to test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714790013


   **[Test build #130179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130179/testReport)** for PR 29874 at commit [`f0ee8b8`](https://github.com/apache/spark/commit/f0ee8b83a84d89f2533e6684b740a623c1d893f2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699604867


   **[Test build #129146 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129146/testReport)** for PR 29874 at commit [`de4c4d7`](https://github.com/apache/spark/commit/de4c4d7d47fbea164edba148ddf2341bd7b065de).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714819284


   Thank you for your first contribution ,@ankits.
   You are added to the Apache Spark contributor group and SPARK-32998 is assigned to you.
   Welcome to the Apache Spark community!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-704616628


   **[Test build #129478 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129478/testReport)** for PR 29874 at commit [`fba96a5`](https://github.com/apache/spark/commit/fba96a577cc485b1e23d22f2f79ed52d256dfe27).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714036852


   **[Test build #130108 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130108/testReport)** for PR 29874 at commit [`ff644eb`](https://github.com/apache/spark/commit/ff644eb0c291a4ede8ac0237ae31a3fd68a22a8e).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-704651035


   **[Test build #129478 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129478/testReport)** for PR 29874 at commit [`fba96a5`](https://github.com/apache/spark/commit/fba96a577cc485b1e23d22f2f79ed52d256dfe27).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709666543






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714038764






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r508095208



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -61,7 +61,8 @@ private[hive] object IsolatedClientLoader extends Logging {
     val files = if (resolvedVersions.contains((resolvedVersion, hadoopVersion))) {
       resolvedVersions((resolvedVersion, hadoopVersion))
     } else {
-      val remoteRepos = sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)
+      val remoteRepos = sys.env.getOrElse(
+        "DEFAULT_ARTIFACT_REPOSITORY", sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))

Review comment:
       @mridulm So it is a bit more complicated than this. We need the options on the Surefire JVM, not the Maven JVM, so it's actually proper to use a flag like `-DargLine="-Dspark.sql.maven.additionalRemoteRepositories=..."`. In my tests this had some negative interaction with the Hive module, which overrides the default `argLine` with a custom one to disable assertions.
   
   Even if that problem can be solved, the Hive test suite launches _entirely separate processes_ via `spark-submit` which are unknown to Maven/Surefire, so even `argLine` is insufficient to pass along configs.
   
   I was able to get it picked up by setting `JAVA_TOOL_OPTIONS="-Dspark...."`, but this affects every single JVM and is a bit messy.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-704651444


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-713890350


   **[Test build #130108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130108/testReport)** for PR 29874 at commit [`ff644eb`](https://github.com/apache/spark/commit/ff644eb0c291a4ede8ac0237ae31a3fd68a22a8e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714843015


   > Building spark internally ...
   
   > Do we need to change `pom.xml` and `project/SparkBuild.scala`?
   
   > I tested master internally just now and didn't need to update `pom.xml` or `project/SparkBuild.scala`.
   
   I could not follow why we don't need to update `pom.xml` and `project/SparkBuild.scala`. Isn't this PR to provide an option for a mirror that should be used during building? Or does this PR target to set a mirror to use during runtime e.g. Spark summit and configuration?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702985437


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129371/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699109379


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r508095208



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -61,7 +61,8 @@ private[hive] object IsolatedClientLoader extends Logging {
     val files = if (resolvedVersions.contains((resolvedVersion, hadoopVersion))) {
       resolvedVersions((resolvedVersion, hadoopVersion))
     } else {
-      val remoteRepos = sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)
+      val remoteRepos = sys.env.getOrElse(
+        "DEFAULT_ARTIFACT_REPOSITORY", sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))

Review comment:
       @mridulm So it is a bit more complicated than this. We need the options on the Surefire JVM, not the Maven JVM, so it's actually proper to use a flag like `-DargLine="-Dspark.sql.maven.additionalRemoteRepositories=..."`. In my tests this had some negative interaction with the Hive module, which overrides the default `argLine` with a custom one to disable assertions.
   
   Even if that problem can be solved, the Hive test suite launches _entirely separate processes_ via `spark-submit`, so even `argLine` is insufficient to pass along configs.
   
   I was able to get it picked up by setting `JAVA_TOOL_OPTIONS="-Dspark...."`, but this affects every single JVM and is a bit messy.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709644031


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34465/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709665955


   **[Test build #129859 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129859/testReport)** for PR 29874 at commit [`8da6fd5`](https://github.com/apache/spark/commit/8da6fd50dc3ef4c36e470b74b963476f064a43f2).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r495202070



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2511,7 +2511,8 @@ object SQLConf {
       .version("3.0.0")
       .stringConf
       .createWithDefault(
-        "https://maven-central.storage-download.googleapis.com/maven2/")
+        sys.env.getOrElse(
+          "INTERNAL_ARTIFACTORY", "https://maven-central.storage-download.googleapis.com/maven2/"))

Review comment:
       I don't think it makes sense to use the environment variable here. This is already configurable and just set as a fallback. If anything, the code in `IsolatedClientLoader` could be updated to leverage the environment variable since, as mentioned in the config doc, Maven Central will still be tried first even if this is configured to be something else.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714903561


   Ah, right. That makes sense.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-712564310






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709666549


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129859/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-712513639






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714841081






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699618359


   **[Test build #129146 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129146/testReport)** for PR 29874 at commit [`de4c4d7`](https://github.com/apache/spark/commit/de4c4d7d47fbea164edba148ddf2341bd7b065de).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-704651448


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129478/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709630822


   Retest this please


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699109379


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-706344932


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129590/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] mridulm commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r505995320



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -61,7 +61,8 @@ private[hive] object IsolatedClientLoader extends Logging {
     val files = if (resolvedVersions.contains((resolvedVersion, hadoopVersion))) {
       resolvedVersions((resolvedVersion, hadoopVersion))
     } else {
-      val remoteRepos = sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)
+      val remoteRepos = sys.env.getOrElse(
+        "DEFAULT_ARTIFACT_REPOSITORY", sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))

Review comment:
       `MAVEN_OPTS` should do the trick, no ?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714833305






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709631946


   **[Test build #129859 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129859/testReport)** for PR 29874 at commit [`8da6fd5`](https://github.com/apache/spark/commit/8da6fd50dc3ef4c36e470b74b963476f064a43f2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r510481629



##########
File path: build/sbt-launch-lib.bash
##########
@@ -39,7 +39,13 @@ dlog () {
 
 acquire_sbt_jar () {
   SBT_VERSION=`awk -F "=" '/sbt\.version/ {print $2}' ./project/build.properties`
-  URL1=https://repo1.maven.org/maven2/org/scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch-${SBT_VERSION}.jar
+  #
+  # TYPESAFE_MIRROR and DEFAULT_ARTIFACT_REPOSITORY env variables can be used to only fetch
+  # artifacts from internal repos only.
+  # Ex:
+  #   TYPESAFE_MIRROR=https://artifacts.internal.com/lightbend-dist

Review comment:
       `TYPESAFE_MIRROR` is only used at `build/mvn`
   ```
   $ git grep TYPESAFE_MIRROR
   build/mvn:    local TYPESAFE_MIRROR=${TYPESAFE_MIRROR:-https://downloads.lightbend.com}
   build/mvn:      "${TYPESAFE_MIRROR}/zinc/${ZINC_VERSION}" \
   build/mvn:  local TYPESAFE_MIRROR=${TYPESAFE_MIRROR:-https://downloads.lightbend.com}
   build/mvn:    "${TYPESAFE_MIRROR}/scala/${scala_version}" \
   ```
   
   Shall we skip to mention it here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-712471695


   **[Test build #130023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130023/testReport)** for PR 29874 at commit [`11a9a67`](https://github.com/apache/spark/commit/11a9a67771c6e67153bb4ea9548a2921dd2af7ca).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714784073


   **[Test build #130177 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130177/testReport)** for PR 29874 at commit [`5a90eb7`](https://github.com/apache/spark/commit/5a90eb7974614edd961a946be3c93b28160a2136).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun closed pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #29874:
URL: https://github.com/apache/spark/pull/29874


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714818691


   Thank you, @ankits and @srowen .
   Merged to master for Apache Spark 3.1.0.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] ankits commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
ankits commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r499065894



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -61,7 +61,8 @@ private[hive] object IsolatedClientLoader extends Logging {
     val files = if (resolvedVersions.contains((resolvedVersion, hadoopVersion))) {
       resolvedVersions((resolvedVersion, hadoopVersion))
     } else {
-      val remoteRepos = sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)
+      val remoteRepos = sys.env.getOrElse(
+        "DEFAULT_ARTIFACT_REPOSITORY", sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))

Review comment:
       @xkrogen During my testing of your suggested changes, the test still tries to download the artifact from `SQLConf.ADDITIONAL_REMOTE_REPOSITORIES` which points to `https://maven-central.storage-download.googleapis.com/maven2/`. I still need the change in `SQLConf.scala` to overwrite the maven repo. 
   
   ```   val ADDITIONAL_REMOTE_REPOSITORIES =
       buildConf("spark.sql.maven.additionalRemoteRepositories")
         .doc("A comma-delimited string config of the optional additional remote Maven mirror " +
           "repositories. This is only used for downloading Hive jars in IsolatedClientLoader " +
           "if the default Maven Central repo is unreachable.")
         .version("3.0.0")
         .stringConf
         .createWithDefault(
           sys.env.getOrElse(
             "DEFAULT_ARTIFACT_REPOSITORY",
             "https://maven-central.storage-download.googleapis.com/maven2/"))```
   
   Let me know your thoughts on this. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-712513336


   **[Test build #130023 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130023/testReport)** for PR 29874 at commit [`11a9a67`](https://github.com/apache/spark/commit/11a9a67771c6e67153bb4ea9548a2921dd2af7ca).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702984982


   **[Test build #129371 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129371/testReport)** for PR 29874 at commit [`12a06c0`](https://github.com/apache/spark/commit/12a06c042011ef8302ab2b61c935714c58e8453f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r495992294



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -61,7 +61,8 @@ private[hive] object IsolatedClientLoader extends Logging {
     val files = if (resolvedVersions.contains((resolvedVersion, hadoopVersion))) {
       resolvedVersions((resolvedVersion, hadoopVersion))
     } else {
-      val remoteRepos = sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)
+      val remoteRepos = sys.env.getOrElse(
+        "DEFAULT_ARTIFACT_REPOSITORY", sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))

Review comment:
       This still isn't quite right -- `remoteRepos` still represents the repositories to be tried _after_ the default Maven Central repository. I looked more closely at this and actually the changes you've made in `SparkSubmit` should already cover this scenario. `IsolatedClientLoader#downloadVersion()` calls `SparkSubmitUtils.resolveMavenCoordinates()` using `SparkSubmitUtils.buildIvySettings()`, which in turn calls `SparkSubmitUtils.createRepoResolvers()` to get the default repository, which is where you've already made your changes.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon edited a comment on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714843015


   > Building spark internally ...
   
   > Do we need to change `pom.xml` and `project/SparkBuild.scala`?
   
   > I tested master internally just now and didn't need to update `pom.xml` or `project/SparkBuild.scala`.
   
   I could not follow why we don't need to update `pom.xml` and `project/SparkBuild.scala`. Isn't this PR to provide an option for a mirror that should be used during building? Or does this PR target to set a mirror to use during runtime e.g. Spark submit and configuration?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714840545


   **[Test build #130179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130179/testReport)** for PR 29874 at commit [`f0ee8b8`](https://github.com/apache/spark/commit/f0ee8b83a84d89f2533e6684b740a623c1d893f2).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-706290156


   **[Test build #129590 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129590/testReport)** for PR 29874 at commit [`8da6fd5`](https://github.com/apache/spark/commit/8da6fd50dc3ef4c36e470b74b963476f064a43f2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-706344921






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709648532






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702977127


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702976554


   **[Test build #129369 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129369/testReport)** for PR 29874 at commit [`1b0ba28`](https://github.com/apache/spark/commit/1b0ba28af9e3c7b80ebc095bea8b78b70c5b5c4a).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709666543


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709631280


   Unfortunately, this PR doesn't pass both Jenkins and Github Action until now. 
   @ankits . Is this tested in your environment with unit tests and integration tests before?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702977132


   Test FAILed.
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/129369/
   Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-702984982


   **[Test build #129371 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129371/testReport)** for PR 29874 at commit [`12a06c0`](https://github.com/apache/spark/commit/12a06c042011ef8302ab2b61c935714c58e8453f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-706344388


   **[Test build #129590 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129590/testReport)** for PR 29874 at commit [`8da6fd5`](https://github.com/apache/spark/commit/8da6fd50dc3ef4c36e470b74b963476f064a43f2).
    * This patch **fails Spark unit tests**.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-713915482






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-713886427


   Retest this please.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r506547330



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -61,7 +61,8 @@ private[hive] object IsolatedClientLoader extends Logging {
     val files = if (resolvedVersions.contains((resolvedVersion, hadoopVersion))) {
       resolvedVersions((resolvedVersion, hadoopVersion))
     } else {
-      val remoteRepos = sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)
+      val remoteRepos = sys.env.getOrElse(
+        "DEFAULT_ARTIFACT_REPOSITORY", sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))

Review comment:
       Yeah it seems so @mridulm -- after I wrote my last comment I started to wonder if that was correct, then saw in the documentation that Spark will pick up config values from System properties. Still trying to make it work in my environment, but I think you are right.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714659678


   **[Test build #130171 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130171/testReport)** for PR 29874 at commit [`b58631c`](https://github.com/apache/spark/commit/b58631c9414b74444fb20921cee56fa02bf3cae7).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699618497






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714763850


   **[Test build #130171 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130171/testReport)** for PR 29874 at commit [`b58631c`](https://github.com/apache/spark/commit/b58631c9414b74444fb20921cee56fa02bf3cae7).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699108878


   Can one of the admins verify this patch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714843399






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714843399






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r510266462



##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -1160,13 +1160,16 @@ private[spark] object SparkSubmitUtils {
     val br: IBiblioResolver = new IBiblioResolver
     br.setM2compatible(true)
     br.setUsepoms(true)
+    val defaultInternalRepo : Option[String] = sys.env.get("DEFAULT_ARTIFACT_REPOSITORY")
+    br.setRoot(defaultInternalRepo.getOrElse("https://repo1.maven.org/maven2/"))

Review comment:
       Is the default value always `https://repo1.maven.org/maven2/` for all user environments?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r510264520



##########
File path: build/sbt-launch-lib.bash
##########
@@ -39,7 +39,20 @@ dlog () {
 
 acquire_sbt_jar () {
   SBT_VERSION=`awk -F "=" '/sbt\.version/ {print $2}' ./project/build.properties`
-  URL1=https://repo1.maven.org/maven2/org/scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch-${SBT_VERSION}.jar
+  # If internal TYPESAFE_SBT_MIRROR is set in the env variable then use that url instead of bintray.
+  # Ex:
+  #  TYPESAFE_SBT_MIRROR=https://artifacts.internal.com/typesafe-ivy-releases-cache
+  #
+  # Similarly, TYPESAVE_MIRROR and DEFAULT_ARTIFACT_REPOSITORY env variables can be used to only fetch

Review comment:
       `TYPESAVE_MIRROR` -> `TYPESAFE_MIRROR`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-712518103


   **[Test build #130028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130028/testReport)** for PR 29874 at commit [`ff644eb`](https://github.com/apache/spark/commit/ff644eb0c291a4ede8ac0237ae31a3fd68a22a8e).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998][BUILD] Add ability to override default remote repos with internal one

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-714792987


   **[Test build #130180 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/130180/testReport)** for PR 29874 at commit [`cfb86c3`](https://github.com/apache/spark/commit/cfb86c3608481b8767ad6d104ce3a2884388c13d).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-713915472


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34717/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on a change in pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r499067150



##########
File path: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -61,7 +61,8 @@ private[hive] object IsolatedClientLoader extends Logging {
     val files = if (resolvedVersions.contains((resolvedVersion, hadoopVersion))) {
       resolvedVersions((resolvedVersion, hadoopVersion))
     } else {
-      val remoteRepos = sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)
+      val remoteRepos = sys.env.getOrElse(
+        "DEFAULT_ARTIFACT_REPOSITORY", sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))

Review comment:
       IMO if you want to fully change the repository, you should be configuring both `DEFAULT_ARTIFACT_REPOSITORY` and `spark.sql.maven.additionalRemoteRepositories`. I think having a config whose default value changes based on an environment variable is confusing behavior.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
HyukjinKwon removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-699604684


   ok to test


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709648532


   Merged build finished. Test FAILed.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-704616628


   **[Test build #129478 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/129478/testReport)** for PR 29874 at commit [`fba96a5`](https://github.com/apache/spark/commit/fba96a577cc485b1e23d22f2f79ed52d256dfe27).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #29874: [SPARK-32998] Add ability to override default remote repos with inter…

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29874:
URL: https://github.com/apache/spark/pull/29874#issuecomment-709648524


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34465/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org