You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2025/02/20 02:32:27 UTC
(spark) branch master updated: [SPARK-51254][PYTHON][CONNECT] Disallow --master with Spark Connect URL
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6603a4ef75eb [SPARK-51254][PYTHON][CONNECT] Disallow --master with Spark Connect URL
6603a4ef75eb is described below
commit 6603a4ef75eb94513b3ff4c98b2e243f38103fe2
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Thu Feb 20 11:32:15 2025 +0900
[SPARK-51254][PYTHON][CONNECT] Disallow --master with Spark Connect URL
### What changes were proposed in this pull request?
This PR proposes to disallow Spark Connect strings in `--master` when Spark API mode is `connect`. This is Python specific issue.
### Why are the changes needed?
Should work as documented in https://github.com/apache/spark/pull/49107
### Does this PR introduce _any_ user-facing change?
Not yet because the main change has not been released (https://github.com/apache/spark/pull/49107)
### How was this patch tested?
Manually tested:
```
./bin/pyspark --master "sc://localhost:15002" --conf spark.api.mode=connect
```
```
Python 3.11.9 (main, Apr 19 2024, 11:44:45) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
/.../spark/python/pyspark/shell.py:77: UserWarning: Failed to initialize Spark session.
warnings.warn("Failed to initialize Spark session.")
Traceback (most recent call last):
File "/.../spark/python/pyspark/shell.py", line 52, in <module>
spark = SparkSession.builder.getOrCreate()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/.../spark/python/pyspark/sql/session.py", line 512, in getOrCreate
raise PySparkRuntimeError(
pyspark.errors.exceptions.base.PySparkRuntimeError: [MASTER_URL_INVALID] Master must either be yarn or start with spark, k8s, or local.
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50000 from HyukjinKwon/SPARK-51254.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/errors/error-conditions.json | 5 +++++
python/pyspark/sql/session.py | 5 +++++
2 files changed, 10 insertions(+)
diff --git a/python/pyspark/errors/error-conditions.json b/python/pyspark/errors/error-conditions.json
index 2a3bcd7240e7..33730f757dbd 100644
--- a/python/pyspark/errors/error-conditions.json
+++ b/python/pyspark/errors/error-conditions.json
@@ -507,6 +507,11 @@
"Variant binary is malformed. Please check the data source is valid."
]
},
+ "MASTER_URL_INVALID": {
+ "message": [
+ "Master must either be yarn or start with spark, k8s, or local."
+ ]
+ },
"MASTER_URL_NOT_SET": {
"message": [
"A master URL must be set in your configuration."
diff --git a/python/pyspark/sql/session.py b/python/pyspark/sql/session.py
index 11bd1ded42b9..241424c954b2 100644
--- a/python/pyspark/sql/session.py
+++ b/python/pyspark/sql/session.py
@@ -508,6 +508,11 @@ class SparkSession(SparkConversionMixin):
if url is None and is_api_mode_connect:
url = opts.get("spark.master", os.environ.get("MASTER", "local"))
+ if url.startswith("sc://"):
+ raise PySparkRuntimeError(
+ errorClass="MASTER_URL_INVALID",
+ messageParameters={},
+ )
if url is None:
raise PySparkRuntimeError(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org