You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ravi Dalal (Jira)" <ji...@apache.org> on 2024/05/24 20:42:00 UTC
[jira] [Updated] (SPARK-48417) Filesystems do not load with spark.jars.packages configuration
[ https://issues.apache.org/jira/browse/SPARK-48417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Dalal updated SPARK-48417:
-------------------------------
Attachment: pyspark_mleap.py
pyspark_spark_jar_package_config_logs.txt
pyspark_without_spark_jar_package_config_logs.txt
> Filesystems do not load with spark.jars.packages configuration
> --------------------------------------------------------------
>
> Key: SPARK-48417
> URL: https://issues.apache.org/jira/browse/SPARK-48417
> Project: Spark
> Issue Type: Bug
> Components: Input/Output
> Affects Versions: 3.5.1
> Reporter: Ravi Dalal
> Priority: Major
> Attachments: pyspark_mleap.py, pyspark_spark_jar_package_config_logs.txt, pyspark_without_spark_jar_package_config_logs.txt
>
>
> When we use spark.jars.packages configuration parameter in Python SparkSession Builder (Pyspark), it appears that the filesystems are not loaded when session starts. Because of this, Spark fails to read file from Google Cloud Storage (GCS) bucket (with GCS Connector).
> I tested this with different packages so it does not appear specific to a particular package. I will attach the sample code and debug logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org