You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "qianlan717 (via GitHub)" <gi...@apache.org> on 2026/10/02 23:58:05 UTC
[PR] [SQL] Only add explicitly configured remote repositories with custom Ivy settings [spark]
qianlan717 opened a new pull request, #59220:
URL: https://github.com/apache/spark/pull/59220
### What changes were proposed in this pull request?\n\nThis PR changes Hive metastore dependency resolution when a custom spark.jars.ivySettings file is configured:\n\n- Add spark.sql.maven.additionalRemoteRepositories only when the user explicitly sets it.\n- Preserve the existing effective default repository behavior when no custom Ivy settings file is used.\n- Document the custom Ivy settings behavior.\n- Add regression coverage for default and explicitly configured repositories.\n\n### Why are the changes needed?\n\nspark.sql.maven.additionalRemoteRepositories has a default Google Cloud Storage Maven mirror. IsolatedClientLoader previously read the effective value unconditionally and appended that mirror even when users supplied a custom Ivy settings file.\n\nUsers who provide custom Ivy settings expect those settings to control repository resolution. Implicitly adding another repository can violate network or repository policy and causes unexpected outbound access.\n\n### Do
es this PR introduce _any_ user-facing change?\n\nYes.\n\nPreviously, setting spark.jars.ivySettings implicitly appended the default spark.sql.maven.additionalRemoteRepositories value, including the default Google Cloud Storage mirror.\n\nAfter this change, a custom Ivy settings file is used without additional remote repositories unless spark.sql.maven.additionalRemoteRepositories is explicitly configured. Behavior without a custom Ivy settings file remains unchanged.\n\n### How was this patch tested?\n\nAdded assertions to IsolatedClientLoaderIvySettingsSuite covering:\n\n- Custom Ivy settings with no explicit additional repositories.\n- Resolution without custom Ivy settings, which retains the effective default.\n- Explicit additional repositories with and without custom Ivy settings.\n\nRan:\n\n build/sbt 'hive/testOnly org.apache.spark.sql.hive.client.IsolatedClientLoaderIvySettingsSuite'\n\nResult: 2 tests passed, 0 failed.\n\n### Was this patch authored or co-authored using
generative AI tooling?\n\nGenerated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59966][SQL] Only add explicitly configured remote repositories with custom Ivy settings [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #59220:
URL: https://github.com/apache/spark/pull/59220#issuecomment-6014535642
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/7d30a7e0e047040b4bf2ebd864fe23bdfd9a1de2
- merged into branch-4.x https://github.com/apache/spark/commit/8afa4418ad16162a310d1d3d81ad66a57790b95e
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59966][SQL] Only add explicitly configured remote repositories with custom Ivy settings [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #59220:
URL: https://github.com/apache/spark/pull/59220#discussion_r4187202327
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala:
##########
@@ -122,12 +122,22 @@ private[hive] object IsolatedClientLoader extends Logging {
}
}
+ private[hive] def getRemoteRepos(
+ sparkConf: SparkConf,
+ ivySettingsPath: Option[String]): Option[String] = {
+ if (ivySettingsPath.isDefined) {
+ sparkConf.getOption(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES.key)
+ } else {
Review Comment:
The two branches now read the same conf in different ways:
- The default branch (and the pre-PR code on both branches) uses `sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)`. That goes through `ConfigReader`, which expands `${env:...}`, `${system:...}` and `${spark.other.key}` references (`ConfigEntryWithDefaultString.readFrom` -> `ConfigReader.get` = `conf.get(key).map(substitute)`).
- The custom-settings branch uses `sparkConf.getOption(key)`, which returns the raw value (`Option(settings.get(key))`) with no substitution.
So, with `spark.jars.ivySettings` set, a value such as `${env:MAVEN_MIRROR_URL}` that resolved correctly before this PR now reaches Ivy as a literal repository root. Fix: use presence only as the gate and keep the typed read:
```
if (ivySettingsPath.isEmpty || sparkConf.contains(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))
Some(sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)) else None
```
`SparkConf.contains(entry)` is public and this conf has no alternative keys, so the presence check behaves the same and all four new assertions still pass. A fifth test case with a `${spark.some.key}` value would lock it in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59966][SQL] Only add explicitly configured remote repositories with custom Ivy settings [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #59220: [SPARK-59966][SQL] Only add explicitly configured remote repositories with custom Ivy settings
URL: https://github.com/apache/spark/pull/59220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59966][SQL] Only add explicitly configured remote repositories with custom Ivy settings [spark]
Posted by "qianlan717 (via GitHub)" <gi...@apache.org>.
qianlan717 commented on PR #59220:
URL: https://github.com/apache/spark/pull/59220#issuecomment-5975919050
> Let;s file a JIRA
@HyukjinKwon I filed the JIRA, forgot to link them together. Added it now. Can you help merge the PR to master and 4.x? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59966][SQL] Only add explicitly configured remote repositories with custom Ivy settings [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #59220:
URL: https://github.com/apache/spark/pull/59220#issuecomment-6014522773
Thank you @qianlan717 and @HyukjinKwon!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org