You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sandy Ryza (Jira)" <ji...@apache.org> on 2025/07/17 20:34:00 UTC

[jira] [Created] (SPARK-52854) Prevent setting catalog and database on the session within Pipelines Python definition files

Sandy Ryza created SPARK-52854:
----------------------------------

             Summary: Prevent setting catalog and database on the session within Pipelines Python definition files
                 Key: SPARK-52854
                 URL: https://issues.apache.org/jira/browse/SPARK-52854
             Project: Spark
          Issue Type: Improvement
          Components: Declarative Pipelines
    Affects Versions: 4.1.0
            Reporter: Sandy Ryza


Setting the spark session default catalog and database is an imperative construct that can cause friction and unexpected behavior from within a pipeline declaration. E.g. it makes pipeline behavior sensitive to the order that Python files are imported in, which can be unpredictable. There are already existing mechanisms for setting Spark catalog and database for pipelines:
 * The catalog and database settings in the pipeline spec
 * The name argument on the dataset decorators accepts a fully-qualified name

Raising an error when someone tries to invoke to set a catalog or database in this situation would avoid this unpredictable behavior.

 

The ways to set the catalog and database from Python are:
 * spark.catalog.setCurrentCatalog
 * spark.sql("USE CATALOG")
 * spark.catalog.setCurrentDatabase
 * spark.sql("USE DATABASE")

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org