You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2025/05/08 03:22:10 UTC
(spark) branch branch-4.0 updated: [SPARK-52026][PS][4.0] Block pandas API on Spark on ANSI mode by default
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 711c57852cdc [SPARK-52026][PS][4.0] Block pandas API on Spark on ANSI mode by default
711c57852cdc is described below
commit 711c57852cdc5f92a2df86599b29c12616a19747
Author: Takuya Ueshin <ue...@databricks.com>
AuthorDate: Thu May 8 12:21:58 2025 +0900
[SPARK-52026][PS][4.0] Block pandas API on Spark on ANSI mode by default
### What changes were proposed in this pull request?
This is a backport of #50816.
Blocks pandas API on Spark on ANSI mode by default.
To force it work on ANSI mode, set pandas-on-spark option `compute.fail_on_ansi_mode` to `False`.
```py
>>> spark.conf.get('spark.sql.ansi.enabled')
'true'
>>> import pyspark.pandas as ps
>>> ps.range(1)
Traceback (most recent call last):
...
pyspark.errors.exceptions.base.UnsupportedOperationException: [PANDAS_API_ON_SPARK_FAIL_ON_ANSI_MODE] Pandas API on Spark does not properly work on ANSI mode.
Please set a Spark config 'spark.sql.ansi.enabled' to `false`.
Alternatively set a pandas-on-spark option 'compute.fail_on_ansi_mode' to `False` to force it to work, although it can cause unexpected behavior.
>>> ps.set_option('compute.fail_on_ansi_mode', False)
>>> ps.range(1)
...: PandasAPIOnSparkAdviceWarning: The config 'spark.sql.ansi.enabled' is set to True. This can cause unexpected behavior from pandas API on Spark since pandas API on Spark follows the behavior of pandas, not SQL.
warnings.warn(message, PandasAPIOnSparkAdviceWarning)
id
0 0
>>> spark.conf.set('spark.sql.ansi.enabled', False)
>>> ps.range(1)
id
0 0
```
### Why are the changes needed?
From Spark 4.0, ANSI mode is enabled by default, but pandas API on Spark won't work properly.
To avoid implicitly working wrongly, block it and ask the user how it should work.
### Does this PR introduce _any_ user-facing change?
Yes, pandas API on Spark will fail on ANSI mode by default.
### How was this patch tested?
Manually.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50817 from ueshin/issues/SPARK-52026/4.0/fail_on_ansi.
Authored-by: Takuya Ueshin <ue...@databricks.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/docs/Makefile | 1 +
.../source/migration_guide/pyspark_upgrade.rst | 1 +
.../source/tutorial/pandas_on_spark/options.rst | 5 +++++
python/pyspark/errors/error-conditions.json | 7 ++++++
python/pyspark/pandas/config.py | 17 +++++++++++---
.../pandas/tests/connect/test_parity_typedef.py | 3 ++-
.../pyspark/pandas/tests/plot/test_series_plot.py | 3 ++-
python/pyspark/pandas/tests/test_typedef.py | 3 ++-
python/pyspark/pandas/utils.py | 26 ++++++++++++++--------
9 files changed, 51 insertions(+), 15 deletions(-)
diff --git a/python/docs/Makefile b/python/docs/Makefile
index 045b03a1afd1..eba8a3fffbcd 100644
--- a/python/docs/Makefile
+++ b/python/docs/Makefile
@@ -22,6 +22,7 @@ SOURCEDIR ?= source
BUILDDIR ?= build
export PYTHONPATH=$(realpath ..):$(realpath ../lib/py4j-0.10.9.9-src.zip)
+export SPARK_ANSI_SQL_MODE=false
# Put it first so that "make" without argument is like "make help".
help:
diff --git a/python/docs/source/migration_guide/pyspark_upgrade.rst b/python/docs/source/migration_guide/pyspark_upgrade.rst
index 906e40140cef..4a0748551677 100644
--- a/python/docs/source/migration_guide/pyspark_upgrade.rst
+++ b/python/docs/source/migration_guide/pyspark_upgrade.rst
@@ -75,6 +75,7 @@ Upgrading from PySpark 3.5 to 4.0
* In Spark 4.0, ``compute.ops_on_diff_frames`` is on by default. To restore the previous behavior, set ``compute.ops_on_diff_frames`` to ``false``.
* In Spark 4.0, the data type ``YearMonthIntervalType`` in ``DataFrame.collect`` no longer returns the underlying integers. To restore the previous behavior, set ``PYSPARK_YM_INTERVAL_LEGACY`` environment variable to ``1``.
* In Spark 4.0, items other than functions (e.g. ``DataFrame``, ``Column``, ``StructType``) have been removed from the wildcard import ``from pyspark.sql.functions import *``, you should import these items from proper modules (e.g. ``from pyspark.sql import DataFrame, Column``, ``from pyspark.sql.types import StructType``).
+* In Spark 4.0, pandas API on Spark will raise an exception if the underlying Spark is working with ANSI mode enabled that is enabled by default, as it will not work properly with ANSI mode. To make it work, you need to explicitly disable ANSI mode by setting ``spark.sql.ansi.enabled`` to ``false``. Alternatively you can set a pandas-on-spark option ``compute.fail_on_ansi_mode`` to ``False`` to force it to work, although it can cause unexpected behavior.
Upgrading from PySpark 3.3 to 3.4
diff --git a/python/docs/source/tutorial/pandas_on_spark/options.rst b/python/docs/source/tutorial/pandas_on_spark/options.rst
index 91f128eb351a..e169b5fe0196 100644
--- a/python/docs/source/tutorial/pandas_on_spark/options.rst
+++ b/python/docs/source/tutorial/pandas_on_spark/options.rst
@@ -319,6 +319,11 @@ compute.isin_limit 80 'compute.isin_limit' set
better performance.
compute.pandas_fallback False 'compute.pandas_fallback' sets whether or not to
fallback automatically to Pandas' implementation.
+compute.fail_on_ansi_mode True 'compute.fail_on_ansi_mode' sets whether or not work
+ with ANSI mode. If True, pandas API on Spark raises
+ an exception if the underlying Spark is working with
+ ANSI mode enabled; otherwise, it forces to work
+ although it can cause unexpected behavior.
plotting.max_rows 1000 'plotting.max_rows' sets the visual limit on top-n-
based plots such as `plot.bar` and `plot.pie`. If it
is set to 1000, the first 1000 data points will be
diff --git a/python/pyspark/errors/error-conditions.json b/python/pyspark/errors/error-conditions.json
index 33730f757dbd..33bc5c3418e6 100644
--- a/python/pyspark/errors/error-conditions.json
+++ b/python/pyspark/errors/error-conditions.json
@@ -812,6 +812,13 @@
"<package_name> >= <minimum_version> must be installed; however, it was not found."
]
},
+ "PANDAS_API_ON_SPARK_FAIL_ON_ANSI_MODE": {
+ "message": [
+ "Pandas API on Spark does not properly work on ANSI mode.",
+ "Please set a Spark config 'spark.sql.ansi.enabled' to `false`.",
+ "Alternatively set a pandas-on-spark option 'compute.fail_on_ansi_mode' to `False` to force it to work, although it can cause unexpected behavior."
+ ]
+ },
"PANDAS_UDF_OUTPUT_EXCEEDS_INPUT_ROWS" : {
"message": [
"The Pandas SCALAR_ITER UDF outputs more rows than input rows."
diff --git a/python/pyspark/pandas/config.py b/python/pyspark/pandas/config.py
index 64fbd006570e..f585af0a0cf6 100644
--- a/python/pyspark/pandas/config.py
+++ b/python/pyspark/pandas/config.py
@@ -268,6 +268,17 @@ _options: List[Option] = [
default=False,
types=bool,
),
+ Option(
+ key="compute.fail_on_ansi_mode",
+ doc=(
+ "'compute.fail_on_ansi_mode' sets whether or not work with ANSI mode. "
+ "If True, pandas API on Spark raises an exception if the underlying Spark is "
+ "working with ANSI mode enabled; otherwise, it forces to work although it can "
+ "cause unexpected behavior."
+ ),
+ default=True,
+ types=bool,
+ ),
Option(
key="plotting.max_rows",
doc=(
@@ -374,7 +385,7 @@ def get_option(key: str, default: Union[Any, _NoValueType] = _NoValue) -> Any:
if default is _NoValue:
default = _options_dict[key].default
_options_dict[key].validate(default)
- spark_session = default_session()
+ spark_session = default_session(check_ansi_mode=False)
return json.loads(spark_session.conf.get(_key_format(key), default=json.dumps(default)))
@@ -396,7 +407,7 @@ def set_option(key: str, value: Any) -> None:
"""
_check_option(key)
_options_dict[key].validate(value)
- spark_session = default_session()
+ spark_session = default_session(check_ansi_mode=False)
spark_session.conf.set(_key_format(key), json.dumps(value))
@@ -417,7 +428,7 @@ def reset_option(key: str) -> None:
None
"""
_check_option(key)
- default_session().conf.unset(_key_format(key))
+ default_session(check_ansi_mode=False).conf.unset(_key_format(key))
@contextmanager
diff --git a/python/pyspark/pandas/tests/connect/test_parity_typedef.py b/python/pyspark/pandas/tests/connect/test_parity_typedef.py
index 8df36ade9261..bfc556fd42d9 100644
--- a/python/pyspark/pandas/tests/connect/test_parity_typedef.py
+++ b/python/pyspark/pandas/tests/connect/test_parity_typedef.py
@@ -18,9 +18,10 @@ import unittest
from pyspark.pandas.tests.test_typedef import TypeHintTestsMixin
from pyspark.testing.connectutils import ReusedConnectTestCase
+from pyspark.testing.pandasutils import PandasOnSparkTestUtils
-class TypeHintParityTests(TypeHintTestsMixin, ReusedConnectTestCase):
+class TypeHintParityTests(TypeHintTestsMixin, PandasOnSparkTestUtils, ReusedConnectTestCase):
pass
diff --git a/python/pyspark/pandas/tests/plot/test_series_plot.py b/python/pyspark/pandas/tests/plot/test_series_plot.py
index 61d114f37b0e..35c9b443fd87 100644
--- a/python/pyspark/pandas/tests/plot/test_series_plot.py
+++ b/python/pyspark/pandas/tests/plot/test_series_plot.py
@@ -22,6 +22,7 @@ import numpy as np
from pyspark import pandas as ps
from pyspark.pandas.plot import PandasOnSparkPlotAccessor, BoxPlotBase
+from pyspark.testing.pandasutils import PandasOnSparkTestCase
from pyspark.testing.utils import have_plotly, plotly_requirement_message
@@ -83,7 +84,7 @@ class SeriesPlotTestsMixin:
self.assertEqual([50], result["fliers"])
-class SeriesPlotTests(SeriesPlotTestsMixin, unittest.TestCase):
+class SeriesPlotTests(SeriesPlotTestsMixin, PandasOnSparkTestCase):
pass
diff --git a/python/pyspark/pandas/tests/test_typedef.py b/python/pyspark/pandas/tests/test_typedef.py
index cb1917db2074..cac9aaf193a0 100644
--- a/python/pyspark/pandas/tests/test_typedef.py
+++ b/python/pyspark/pandas/tests/test_typedef.py
@@ -54,6 +54,7 @@ from pyspark.pandas.typedef import (
pandas_on_spark_type,
)
from pyspark import pandas as ps
+from pyspark.testing.pandasutils import PandasOnSparkTestCase
class TypeHintTestsMixin:
@@ -431,7 +432,7 @@ class TypeHintTestsMixin:
self.assertEqual(pandas_on_spark_type(extension_dtype), (extension_dtype, spark_type))
-class TypeHintTests(TypeHintTestsMixin, unittest.TestCase):
+class TypeHintTests(TypeHintTestsMixin, PandasOnSparkTestCase):
pass
diff --git a/python/pyspark/pandas/utils.py b/python/pyspark/pandas/utils.py
index e17afa026c5a..2275d438234c 100644
--- a/python/pyspark/pandas/utils.py
+++ b/python/pyspark/pandas/utils.py
@@ -43,7 +43,7 @@ from pandas.api.types import is_list_like # type: ignore[attr-defined]
from pyspark.sql import functions as F, Column, DataFrame as PySparkDataFrame, SparkSession
from pyspark.sql.types import DoubleType
from pyspark.sql.utils import is_remote
-from pyspark.errors import PySparkTypeError
+from pyspark.errors import PySparkTypeError, UnsupportedOperationException
from pyspark import pandas as ps # noqa: F401
from pyspark.pandas._typing import (
Axis,
@@ -476,7 +476,7 @@ def is_testing() -> bool:
return "SPARK_TESTING" in os.environ
-def default_session() -> SparkSession:
+def default_session(*, check_ansi_mode: bool = True) -> SparkSession:
spark = SparkSession.getActiveSession()
if spark is None:
spark = SparkSession.builder.appName("pandas-on-Spark").getOrCreate()
@@ -485,13 +485,21 @@ def default_session() -> SparkSession:
# the behavior of pandas API on Spark follows pandas, not SQL.
if is_testing():
spark.conf.set("spark.sql.ansi.enabled", False)
- if spark.conf.get("spark.sql.ansi.enabled") == "true":
- log_advice(
- "The config 'spark.sql.ansi.enabled' is set to True. "
- "This can cause unexpected behavior "
- "from pandas API on Spark since pandas API on Spark follows "
- "the behavior of pandas, not SQL."
- )
+
+ if check_ansi_mode:
+ if spark.conf.get("spark.sql.ansi.enabled") == "true":
+ if ps.get_option("compute.fail_on_ansi_mode"):
+ raise UnsupportedOperationException(
+ errorClass="PANDAS_API_ON_SPARK_FAIL_ON_ANSI_MODE",
+ messageParameters={},
+ )
+ else:
+ log_advice(
+ "The config 'spark.sql.ansi.enabled' is set to True. "
+ "This can cause unexpected behavior "
+ "from pandas API on Spark since pandas API on Spark follows "
+ "the behavior of pandas, not SQL."
+ )
return spark
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org