You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ben Burnett (Jira)" <ji...@apache.org> on 2025/09/03 21:10:00 UTC
[jira] [Updated] (SPARK-53478) `SparkContext.addFile` doesn't resolve current job artifact state in Spark Local mode
[ https://issues.apache.org/jira/browse/SPARK-53478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ben Burnett updated SPARK-53478:
--------------------------------
Description:
When I use `SparkContext.addFile` in spark local mode, it adds it to the cwd. When I try to access that file inside of a sql planned RDD operation with `SparkFiles.get`, it tries to resolve the path against the job-specific artifact directory (ie ```\{cwd}/userFiles-\{some_uuid}```) and fails to find the actual file.
It seems like this is because the sql planning is setting up the active job artifact state based on the session that the sql is being called on, while directly accessing the SparkContext from a script in spark local doesn't.
This works in all other configurations of spark so obviously not a huge priority. We only run `addFile` in spark local for testing but it'd be nice to support it since it is a regression.
I think the origin is [https://github.com/apache/spark/commit/26330355836f5b2dad9b7bd4c72d9830c7ce6788] since that's where it changed `SparkFiles.get` but not entirely sure
was:
When I use `SparkContext.addFile` in spark local mode, it adds it to the cwd. When I try to access that file inside of a sql planned RDD operation with `SparkFiles.get`, it tries to resolve the path against the job-specific artifact directory (ie `{cwd}/userFiles-{some_uuid}`) and fails to find the actual file.
It seems like this is because the sql planning is setting up the active job artifact state based on the session that the sql is being called on, while directly accessing the SparkContext from a script in spark local doesn't.
This works in all other configurations of spark so obviously not a huge priority. We only run `addFile` in spark local for testing but it'd be nice to support it since it is a regression.
I think the origin is https://github.com/apache/spark/commit/26330355836f5b2dad9b7bd4c72d9830c7ce6788 since that's where it changed `SparkFiles.get` but not entirely sure
> `SparkContext.addFile` doesn't resolve current job artifact state in Spark Local mode
> -------------------------------------------------------------------------------------
>
> Key: SPARK-53478
> URL: https://issues.apache.org/jira/browse/SPARK-53478
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Ben Burnett
> Priority: Minor
>
> When I use `SparkContext.addFile` in spark local mode, it adds it to the cwd. When I try to access that file inside of a sql planned RDD operation with `SparkFiles.get`, it tries to resolve the path against the job-specific artifact directory (ie ```\{cwd}/userFiles-\{some_uuid}```) and fails to find the actual file.
> It seems like this is because the sql planning is setting up the active job artifact state based on the session that the sql is being called on, while directly accessing the SparkContext from a script in spark local doesn't.
> This works in all other configurations of spark so obviously not a huge priority. We only run `addFile` in spark local for testing but it'd be nice to support it since it is a regression.
> I think the origin is [https://github.com/apache/spark/commit/26330355836f5b2dad9b7bd4c72d9830c7ce6788] since that's where it changed `SparkFiles.get` but not entirely sure
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org