You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "jonmio (via GitHub)" <gi...@apache.org> on 2026/01/15 20:22:51 UTC
[PR] Support SparkSession.Builder.create for PySpark Classic [spark]
jonmio opened a new pull request, #53820:
URL: https://github.com/apache/spark/pull/53820
### What changes were proposed in this pull request?
Allow users to call SparkSession.Builder.create from PySpark in Classic Mode so that they can a new session with the provided builder configs without mutating configurations in any existing sessions.
### Why are the changes needed?
Currently, `create` is not supported in PySpark classic. Users can call `getOrCreate` to obtain a session with the provided configurations, but if a session already exists, `getOrCreate` will mutate the existing configurations in the session. We would like to provide an API for users which guarantees creation of a session and does not have side effects on existing sessions.
This change also aligns the behavior between Classic and Connect mode since `create` is already supported in Connect.
### Does this PR introduce _any_ user-facing change?
Previously calling `SparkSessionBuilder.create()` would throw an exception saying that create is only supported in Connect mode.
After this change, `SparkSessionBuilder.create()` will create a new SparkSession with the specified builder configurations.
### How was this patch tested?
Added the new test class `SparkSessionBuilderCreateTests` in `python/pyspark/sql/tests/test_session.py`
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55055][PYTHON] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #53820:
URL: https://github.com/apache/spark/pull/53820#discussion_r2702968888
##########
python/pyspark/sql/session.py:
##########
@@ -598,11 +601,24 @@ def create(self) -> "SparkSession":
os.environ["SPARK_CONNECT_MODE_ENABLED"] = "1"
opts["spark.remote"] = url
return cast(SparkSession, RemoteSparkSession.builder.config(map=opts).create())
+ # Classic mode
else:
- raise PySparkRuntimeError(
- errorClass="ONLY_SUPPORTED_WITH_SPARK_CONNECT",
- messageParameters={"feature": "SparkSession.builder.create"},
- )
+ with self._lock:
+ # Build SparkConf from options
+ sparkConf = SparkConf()
+ for key, value in self._options.items():
+ sparkConf.set(key, str(value))
+
+ sc = (
+ self._sc
+ if getattr(self, "_sc", None) is not None and self._sc._jsc is not None
Review Comment:
It's set here:
https://github.com/apache/spark/blob/70a3ab5daa4831935748867239549900815c9651/python/pyspark/sql/session.py#L616-L622
in the constructor of `SparkSession`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55055] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "gatorsmile (via GitHub)" <gi...@apache.org>.
gatorsmile commented on code in PR #53820:
URL: https://github.com/apache/spark/pull/53820#discussion_r2699713706
##########
python/pyspark/sql/session.py:
##########
@@ -598,11 +601,24 @@ def create(self) -> "SparkSession":
os.environ["SPARK_CONNECT_MODE_ENABLED"] = "1"
opts["spark.remote"] = url
return cast(SparkSession, RemoteSparkSession.builder.config(map=opts).create())
+ # Classic mode
else:
- raise PySparkRuntimeError(
- errorClass="ONLY_SUPPORTED_WITH_SPARK_CONNECT",
- messageParameters={"feature": "SparkSession.builder.create"},
- )
+ with self._lock:
+ # Build SparkConf from options
+ sparkConf = SparkConf()
+ for key, value in self._options.items():
+ sparkConf.set(key, str(value))
+
+ sc = (
+ self._sc
+ if getattr(self, "_sc", None) is not None and self._sc._jsc is not None
Review Comment:
In the Class Builder, which function actually sets self._sc? Is it None always?
CC @HyukjinKwon and @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55055][PYTHON] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on PR #53820:
URL: https://github.com/apache/spark/pull/53820#issuecomment-3806769975
Planning to merge this this afternoon if the build passes and there are no further concerns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55055][PYTHON] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza commented on PR #53820:
URL: https://github.com/apache/spark/pull/53820#issuecomment-3808163229
Merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55055] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "gatorsmile (via GitHub)" <gi...@apache.org>.
gatorsmile commented on code in PR #53820:
URL: https://github.com/apache/spark/pull/53820#discussion_r2699729030
##########
python/pyspark/sql/tests/test_session.py:
##########
@@ -461,6 +462,114 @@ def test_invalid_create(self):
messageParameters={},
)
+class SparkSessionBuilderCreateTests(unittest.TestCase, PySparkErrorTestUtils):
Review Comment:
Thank you for adding the tests!
Could you help clarify coverage for the following areas?
Error Handling Tests
• How does create() behave when called with invalid configurations?
• What is the expected behavior if SparkContext creation fails?
• How should conflicting SparkContext-level configurations be handled when a context already exists?
Thread Safety Tests
• How should the implementation behave when multiple threads call create() concurrently?
• Are there any expected race conditions between create() and getOrCreate() that we should guard against or test explicitly?
Default/Active Session Update Logic
The PR description states:
“This method will update the default and/or active session if they are not set.”
Could we clarify:
• What is the expected behavior for updating the default session, and how should this be tested?
• In what cases should the default session be updated or preserved?
• Are there additional scenarios around active session updates that should be validated?
Session Stop/Cleanup
• What should happen when stop() is called on a session created via create()?
• How can we ensure resources are properly cleaned up?
• Should stopping one session affect other sessions sharing the same SparkContext?
Connect Mode Fallback
• How do we want to validate that the existing Connect mode logic continues to work correctly after these changes?
appName Configuration
• Should we explicitly verify that appName() and other common builder configurations work correctly with create() in addition to master()?
Catalog/Database Operations
• What expectations do we have around catalog operations for sessions created via create()? For example:
• Creating databases or tables
• Cross-session visibility of catalogs
Duplicate Config Keys
• What should the expected behavior be if the same config key is set multiple times in the builder before calling create()?
Remote Mode Error Paths
• Are there specific error scenarios in the Connect/remote path (e.g., invalid remote URLs) that we should explicitly test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55055][PYTHON] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "jonmio (via GitHub)" <gi...@apache.org>.
jonmio commented on code in PR #53820:
URL: https://github.com/apache/spark/pull/53820#discussion_r2725883575
##########
python/pyspark/sql/tests/test_session.py:
##########
@@ -461,6 +462,114 @@ def test_invalid_create(self):
messageParameters={},
)
+class SparkSessionBuilderCreateTests(unittest.TestCase, PySparkErrorTestUtils):
Review Comment:
Added some additional tests for the error handling and common confs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55055][PYTHON] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #53820:
URL: https://github.com/apache/spark/pull/53820#discussion_r2703029265
##########
python/pyspark/sql/session.py:
##########
@@ -563,14 +563,14 @@ def getOrCreate(self) -> "SparkSession":
module.applyModifiableSettings(session._jsparkSession, self._options)
return session
- # Spark Connect-specific API
- @remote_only
def create(self) -> "SparkSession":
Review Comment:
in the scala side, `def create` and `def shareOrCreate` share the code by using a `forceCreate` flag, shall we do the same at the python side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55055][PYTHON] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "jonmio (via GitHub)" <gi...@apache.org>.
jonmio commented on code in PR #53820:
URL: https://github.com/apache/spark/pull/53820#discussion_r2725838922
##########
python/pyspark/sql/session.py:
##########
@@ -563,14 +563,14 @@ def getOrCreate(self) -> "SparkSession":
module.applyModifiableSettings(session._jsparkSession, self._options)
return session
- # Spark Connect-specific API
- @remote_only
def create(self) -> "SparkSession":
Review Comment:
I think the `build(foreCreate: Boolean)` pattern works really well on the Scala side within classic mode, but for the Python code we have to handle both connect and classic and it seems like there is a bunch of complex handling for each mode. IMO it would be even more confusing to put this all under one shared function. Lmk if you have good ideas on how to make this code easier to reason about though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #53820:
URL: https://github.com/apache/spark/pull/53820#issuecomment-3756711049
## ⚠️ Pull Request Title Validation
This pull request title does not contain a JIRA issue ID.
Please update the title to either:
- Include a JIRA ID: `[SPARK-12345] Your description`
- Mark as minor change: `[MINOR] Your description`
For minor changes that don't require a JIRA ticket (e.g., typo fixes), please prefix the title with `[MINOR]`.
---
*This comment was automatically generated by GitHub Actions*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55055][PYTHON] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "jonmio (via GitHub)" <gi...@apache.org>.
jonmio commented on code in PR #53820:
URL: https://github.com/apache/spark/pull/53820#discussion_r2725883000
##########
python/pyspark/sql/tests/test_session.py:
##########
@@ -461,6 +462,114 @@ def test_invalid_create(self):
messageParameters={},
)
+class SparkSessionBuilderCreateTests(unittest.TestCase, PySparkErrorTestUtils):
Review Comment:
Thanks for the detailed review! I've added some tests to improve coverage, but also wanted to address a few of your high level concerns around connect mode compatibility, error/handling failure modes, and behaviors with multiple sessions. Importantly, this change is introducing parity with Connect (create already exists there and we are leaning on existing connect test coverage to ensure we have no introduced new regressions), multiple sessions attached to the same context already exists via the newSession API, and we are reusing existing primitives used by getOrCreate and newSession (e.g. session construction helpers and locking). Here are my thoughts on some of the more specific comments:
> Connect mode fallback & invalid remote URLs
This PR doesn't touch connect mode at all, and we rely on existing connect test coverage
> Are there race conditions between create() and getOrCreate()?
Both methods use the same self._lock (line 606) and rely on SparkContext.getOrCreate() which has its own locking via SparkContext._lock. No new race conditions introduced here - just reusing the existing thread-safe mechanisms.
> Duplicate config keys
Standard SparkConf behavior - last write wins. Same as getOrCreate(), nothing specific to create().
> Session stop/cleanup:
Stopping a session stops the parent SparkContext. This stops all sessions attached to the SparkContext, but this is not new behavior in Classic.
> Conflicting SparkContext configs
SparkContext.getOrCreate() handles this at line 615. When a SparkContext already exists, new configs apply to the session only, not the shared context. Same behavior as getOrCreate() - there's even a comment about this at line 558: "Do not update SparkConf for existing SparkContext, as it's shared by all sessions."
Let me know if you would like specific tests for any of these behaviors though, happy to add it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55055][PYTHON] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #53820:
URL: https://github.com/apache/spark/pull/53820#issuecomment-3803675943
```
annotations failed mypy checks:
python/pyspark/sql/session.py:617: error: "Builder" has no attribute "_sc" [attr-defined]
python/pyspark/sql/session.py:618: error: "Builder" has no attribute "_sc" [attr-defined]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55055][PYTHON] Support SparkSession.Builder.create for PySpark Classic [spark]
Posted by "sryza (via GitHub)" <gi...@apache.org>.
sryza closed pull request #53820: [SPARK-55055][PYTHON] Support SparkSession.Builder.create for PySpark Classic
URL: https://github.com/apache/spark/pull/53820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org