You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Mario Grgic (Jira)" <ji...@apache.org> on 2025/07/04 12:55:00 UTC
[jira] [Created] (SPARK-52687) pyspark startup script: Parsing of SPARK_CONNECT_MODE is error prone
Mario Grgic created SPARK-52687:
-----------------------------------
Summary: pyspark startup script: Parsing of SPARK_CONNECT_MODE is error prone
Key: SPARK-52687
URL: https://issues.apache.org/jira/browse/SPARK-52687
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 4.0.0
Reporter: Mario Grgic
I am using Apache Spark 4.0.0 and don't have {{SPARK_CONNECT_MODE}} environment variable set. When launching {{pyspark}} a message:
??The environment variable SPARK_CONNECT_MODE has unknown value or pyspark.util package is not available: 0??
is always shown. Even after exporting the value for this environment variable, the message is still shown with the value of the variable instead of 0.
Looking at the pyspark BASH script it looks like the value of the environment variable is calculated:
{{export SPARK_CONNECT_MODE=$($PYSPARK_DRIVER_PYTHON -c "from pyspark.util import spark_connect_mode; print(spark_connect_mode())")}}
which can (at least on my system) can contain newline and spaces (if the environment variable is not set). This is why the check below:
{{if [[ "$SPARK_CONNECT_MODE" != "0" && "$SPARK_CONNECT_MODE" != "1" ]]; then}}
{{ echo "The environment variable SPARK_CONNECT_MODE has unknown value or pyspark.util package is not available: $SPARK_CONNECT_MODE"}}
{{ unset SPARK_CONNECT_MODE}}
{{fi}}
fails. Note however, that this check will fail *always* even if {{SPARK_CONNECT_MODE}} is set to any value other than 0 or 1. So basically, the script will not accept valid values for this variable and will unset the value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org