You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sanford Ryza (Jira)" <ji...@apache.org> on 2025/10/24 16:16:00 UTC
[jira] [Updated] (SPARK-52854) Prevent setCurrentDatabase and setCurrentCatalog within Pipelines Python definition files
[ https://issues.apache.org/jira/browse/SPARK-52854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sanford Ryza updated SPARK-52854:
---------------------------------
Summary: Prevent setCurrentDatabase and setCurrentCatalog within Pipelines Python definition files (was: Prevent setting catalog and database on the session within Pipelines Python definition files)
> Prevent setCurrentDatabase and setCurrentCatalog within Pipelines Python definition files
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-52854
> URL: https://issues.apache.org/jira/browse/SPARK-52854
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.1.0
> Reporter: Sandy Ryza
> Priority: Major
>
> Setting the spark session default catalog and database is an imperative construct that can cause friction and unexpected behavior from within a pipeline declaration. E.g. it makes pipeline behavior sensitive to the order that Python files are imported in, which can be unpredictable. There are already existing mechanisms for setting Spark catalog and database for pipelines:
> * The catalog and database settings in the pipeline spec
> * The name argument on the dataset decorators accepts a fully-qualified name
> Raising an error when someone tries to invoke to set a catalog or database in this situation would avoid this unpredictable behavior.
>
> The ways to set the catalog and database from Python are:
> * spark.catalog.setCurrentCatalog
> * spark.sql("USE CATALOG")
> * spark.catalog.setCurrentDatabase
> * spark.sql("USE DATABASE")
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org