You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2023/05/18 05:35:53 UTC
[spark] branch master updated: [SPARK-43509][PYTHON][CONNECT][FOLLOW-UP] Set SPARK_CONNECT_MODE_ENABLED when running pyspark shell with remote is local
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 32bdf8a5b52 [SPARK-43509][PYTHON][CONNECT][FOLLOW-UP] Set SPARK_CONNECT_MODE_ENABLED when running pyspark shell with remote is local
32bdf8a5b52 is described below
commit 32bdf8a5b52ab1370dd009cd5967042c7ac20446
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Thu May 18 14:35:39 2023 +0900
[SPARK-43509][PYTHON][CONNECT][FOLLOW-UP] Set SPARK_CONNECT_MODE_ENABLED when running pyspark shell with remote is local
### What changes were proposed in this pull request?
This PR is a followup of https://github.com/apache/spark/pull/41013 that sets `SPARK_CONNECT_MODE_ENABLED` when running PySpark shell via `bin/pyspark --remote=local` so it works.
### Why are the changes needed?
It is broken after the PR:
```bash
./bin/pyspark --remote local
```
```python
...
Traceback (most recent call last):
File "/.../python/pyspark/shell.py", line 78, in <module>
sc = spark.sparkContext
File "/.../python/pyspark/sql/connect/session.py", line 567, in sparkContext
raise PySparkNotImplementedError(
pyspark.errors.exceptions.base.PySparkNotImplementedError: [NOT_IMPLEMENTED] sparkContext() is not implemented.
```
### Does this PR introduce _any_ user-facing change?
No to end users.
Yes to the dev as described above.
### How was this patch tested?
Manually tested via `./bin/pyspark --remote local`
Closes #41206 from HyukjinKwon/SPARK-43509-followup.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
.../org/apache/spark/launcher/SparkSubmitCommandBuilder.java | 1 +
python/pyspark/shell.py | 9 +++++----
2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java b/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
index 289eb31db31..9618673b4ce 100644
--- a/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
+++ b/launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java
@@ -363,6 +363,7 @@ class SparkSubmitCommandBuilder extends AbstractCommandBuilder {
}
if (remoteStr != null) {
env.put("SPARK_REMOTE", remoteStr);
+ env.put("SPARK_CONNECT_MODE_ENABLED", "1");
}
if (!isEmpty(pyOpts)) {
diff --git a/python/pyspark/shell.py b/python/pyspark/shell.py
index 86f576a3029..7e2093c1d31 100644
--- a/python/pyspark/shell.py
+++ b/python/pyspark/shell.py
@@ -100,10 +100,11 @@ print(
% (platform.python_version(), platform.python_build()[0], platform.python_build()[1])
)
if is_remote():
- print(
- "Client connected to the Spark Connect server at %s"
- % urlparse(os.environ["SPARK_REMOTE"]).netloc
- )
+ url = os.environ.get("SPARK_REMOTE", None)
+ assert url is not None
+ if url.startswith("local"):
+ url = "sc://localhost" # only for display in the console.
+ print("Client connected to the Spark Connect server at %s" % urlparse(url).netloc)
else:
print("Spark context Web UI available at %s" % (sc.uiWebUrl)) # type: ignore[union-attr]
print(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org