You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sandy Ryza (Jira)" <ji...@apache.org> on 2025/07/17 20:34:00 UTC
[jira] [Created] (SPARK-52854) Prevent setting catalog and database on the session within Pipelines Python definition files
Sandy Ryza created SPARK-52854:
----------------------------------
Summary: Prevent setting catalog and database on the session within Pipelines Python definition files
Key: SPARK-52854
URL: https://issues.apache.org/jira/browse/SPARK-52854
Project: Spark
Issue Type: Improvement
Components: Declarative Pipelines
Affects Versions: 4.1.0
Reporter: Sandy Ryza
Setting the spark session default catalog and database is an imperative construct that can cause friction and unexpected behavior from within a pipeline declaration. E.g. it makes pipeline behavior sensitive to the order that Python files are imported in, which can be unpredictable. There are already existing mechanisms for setting Spark catalog and database for pipelines:
* The catalog and database settings in the pipeline spec
* The name argument on the dataset decorators accepts a fully-qualified name
Raising an error when someone tries to invoke to set a catalog or database in this situation would avoid this unpredictable behavior.
The ways to set the catalog and database from Python are:
* spark.catalog.setCurrentCatalog
* spark.sql("USE CATALOG")
* spark.catalog.setCurrentDatabase
* spark.sql("USE DATABASE")
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org