[jira] [Assigned] (SPARK-56572) Inject the Spark session into python pipeline code

["Gengliang Wang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-56572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gengliang Wang reassigned SPARK-56572:
--------------------------------------

    Assignee: Andreas Neumann

> Inject the Spark session into python pipeline code
> --------------------------------------------------
>
>                 Key: SPARK-56572
>                 URL: https://issues.apache.org/jira/browse/SPARK-56572
>             Project: Spark
>          Issue Type: Improvement
>          Components: Declarative Pipelines
>    Affects Versions: 4.1.0, 4.1.1
>            Reporter: Andreas Neumann
>            Assignee: Andreas Neumann
>            Priority: Major
>              Labels: pull-request-available
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> In Declarative Pipelines, all Python scripts run as separate modules but have to share the same Spark session. This session is created by the framework, and every script needs to bring it into its own scope by declaring
> {code:java}
> spark = SparkSession.active() {code}
> This bears the risk that users might create their own Spark session instead, and that could break dependencies between different Python scripts of the same pipeline. It would be better to inject this session directly into the module, so user code does not need to worry about obtaining it. 
> Also, change `spark-pipelines init` to omit that line from the generated sample code.   



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org