You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Qianlan Chen (Jira)" <ji...@apache.org> on 2026/09/15 22:14:00 UTC
[jira] [Commented] (SPARK-59553) Make SparkSession.addArtifact honor spark.jars.ivySettings
[ https://issues.apache.org/jira/browse/SPARK-59553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115800#comment-18115800 ]
Qianlan Chen commented on SPARK-59553:
--------------------------------------
I'm working on this issue
> Make SparkSession.addArtifact honor spark.jars.ivySettings
> ----------------------------------------------------------
>
> Key: SPARK-59553
> URL: https://issues.apache.org/jira/browse/SPARK-59553
> Project: Spark
> Issue Type: Task
> Components: Connect, SQL
> Affects Versions: 4.2.0
> Environment: SparkSession.addArtifact supports resolving Maven coordinates through ivy:// URIs, but it does not currently honor `spark.jars.ivySettings`.
> For classic Spark, Artifact.newIvyArtifacts always builds the default Ivy settings. For Spark Connect, Ivy coordinates are resolved by the client and the resulting JARs are uploaded to the server. Consequently, the client's Ivy configuration and network access are used instead of the server's spark.jars.ivySettings.
> This differs from other Maven-resolution surfaces such as spark-submit and SparkContext.addJar. It also prevents Spark Connect from using repositories, mirrors, credentials, or repository policies configured only on the server.
> Proposed changes:
> * Make SparkSession.addArtifact and addArtifacts use spark.jars.ivySettings when resolving ivy:// coordinates.
> * Move Spark Connect Ivy resolution from the client to the server:
> * Add a protocol representation for an unresolved Ivy coordinate.
> * Have the client send the coordinate instead of resolving and uploading its JARs.
> * Resolve the coordinate on the server and register the resulting JARs through the existing artifact-management path.
> * Preserve URI ordering and deduplicate shared transitive dependencies.
> * Apply the behavior to both addArtifact and addArtifacts.
> Compatibility and fallback:
> * When spark.jars.ivySettings is configured, load it and use it for resolution.
> * When it is not configured, retain the existing default Ivy resolver behavior.
> * A client should use server-side resolution only when the server advertises support. With an older server, retain the existing client-side resolution and upload behavior.
> * Older clients must continue to work with newer servers.
> * Preserve the existing handling of per-coordinate ?repos= repositories. If these cannot safely be resolved by the server, fall back to the existing client-side path.
> * A failure to load or use an explicitly configured spark.jars.ivySettings should be reported rather than silently bypassing it and resolving from default repositories.
> Reporter: Qianlan Chen
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org