(spark) branch master updated: [SPARK-55658][PYTHON] SparkSessionBuilder.create in PySpark classic should mirror getOrCreate path as much as possible

[we...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new f5ca55cc2707 [SPARK-55658][PYTHON] SparkSessionBuilder.create in PySpark classic should mirror getOrCreate path as much as possible
f5ca55cc2707 is described below

commit f5ca55cc2707473897c1fd34881e29c9ba80cb0b
Author: Jon Mio <16...@users.noreply.github.com>
AuthorDate: Tue May 5 23:23:52 2026 +0800

    [SPARK-55658][PYTHON] SparkSessionBuilder.create in PySpark classic should mirror getOrCreate path as much as possible
    
    ### What changes were proposed in this pull request?
    As titled this is a minor hygiene improvement which brings the create and getOrCreate codepaths closer together. Prior to this PR, the create codepath would always create a new SparkConf and pass that to SparkContext.getOrCreate. When a SparkSession/Context already exists, the creation of the SparkConf is not required at all and the SparkContext can be fetched from the instantiated session.
    
    Note that this change only modifies SparkSessionBuilder.create in PySpark classic which was recently added [here](https://github.com/apache/spark/pull/53820).
    
    ### Why are the changes needed?
    Code hygiene
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    
    We mainly rely on existing tests for the create codepath. We also add a test to verify that a SparkConf is not created when there is an existing session.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No
    
    Closes #54429 from jonmio/jonmio-unify-create-and-getOrCreate.
    
    Authored-by: Jon Mio <16...@users.noreply.github.com>
    Signed-off-by: Wenchen Fan <we...@databricks.com>
---
 python/pyspark/sql/session.py            | 19 +++++++++++++------
 python/pyspark/sql/tests/test_session.py | 26 ++++++++++++++++++++++++++
 2 files changed, 39 insertions(+), 6 deletions(-)

diff --git a/python/pyspark/sql/session.py b/python/pyspark/sql/session.py
index e782f2f79db4..3878af7bd5de 100644
--- a/python/pyspark/sql/session.py
+++ b/python/pyspark/sql/session.py
@@ -607,12 +607,19 @@ class SparkSession(SparkConversionMixin):
                 from pyspark.core.context import SparkContext
 
                 with self._lock:
-                    # Build SparkConf from options
-                    sparkConf = SparkConf()
-                    for key, value in self._options.items():
-                        sparkConf.set(key, str(value))
-
-                    sc = SparkContext.getOrCreate(sparkConf)
+                    instantiated_session = SparkSession._instantiatedSession
+                    # Get SparkContext
+                    if (
+                        instantiated_session is not None
+                        and instantiated_session._sc._jsc is not None
+                    ):
+                        sc = instantiated_session._sc
+                    else:
+                        sparkConf = SparkConf()
+                        for key, value in self._options.items():
+                            sparkConf.set(key, value)
+                        # This SparkContext may be an existing one.
+                        sc = SparkContext.getOrCreate(sparkConf)
                     jSparkSessionClass = SparkSession._get_j_spark_session_class(sc._jvm)
                     # Create a new SparkSession in the JVM
                     jSparkSession = jSparkSessionClass.builder().config(self._options).create()
diff --git a/python/pyspark/sql/tests/test_session.py b/python/pyspark/sql/tests/test_session.py
index 3606056f6793..fb86deb33a2d 100644
--- a/python/pyspark/sql/tests/test_session.py
+++ b/python/pyspark/sql/tests/test_session.py
@@ -616,6 +616,32 @@ class SparkSessionBuilderCreateTests(unittest.TestCase, PySparkErrorTestUtils):
         finally:
             session2.stop()
 
+    def test_create_does_not_construct_spark_conf_when_session_exists(self):
+        """Ensure SparkConf() is not called when a valid session already exists."""
+        self.session = self._get_builder().create()
+        with unittest.mock.patch("pyspark.sql.session.SparkConf") as mock_spark_conf:
+            session2 = self._get_builder().create()
+            try:
+                mock_spark_conf.assert_not_called()
+                self.assertIs(session2.sparkContext, self.session.sparkContext)
+            finally:
+                session2.stop()
+
+    def test_create_applies_mutable_conf_to_second_session(self):
+        """
+        Ensure that mutable SQL configs passed to create() are applied per-session
+        even when a valid SparkSession already exists.
+        """
+        key = "spark.sql.shuffle.partitions"
+        self.session = self._get_builder().config(key, "5").create()
+        self.assertEqual(self.session.conf.get(key), "5")
+        session2 = self._get_builder().config(key, "7").create()
+        try:
+            self.assertEqual(session2.conf.get(key), "7")
+            self.assertIs(session2.sparkContext, self.session.sparkContext)
+        finally:
+            session2.stop()
+
 
 class SparkSessionProfileTests(unittest.TestCase, PySparkErrorTestUtils):
     def setUp(self):


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org