You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2025/05/08 08:59:16 UTC
(spark) branch master updated: [SPARK-52026][PS] Block pandas API on Spark on ANSI mode by default
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 584f4689ae30 [SPARK-52026][PS] Block pandas API on Spark on ANSI mode by default
584f4689ae30 is described below
commit 584f4689ae30f26ac2166fc1d6010d4deb5ba231
Author: Takuya Ueshin <ue...@databricks.com>
AuthorDate: Thu May 8 16:59:00 2025 +0800
[SPARK-52026][PS] Block pandas API on Spark on ANSI mode by default
### What changes were proposed in this pull request?
Blocks pandas API on Spark on ANSI mode by default.
To force it work on ANSI mode, set pandas-on-spark option `compute.fail_on_ansi_mode` to `False`.
```py
>>> spark.conf.get('spark.sql.ansi.enabled')
'true'
>>> import pyspark.pandas as ps
>>> ps.range(1)
Traceback (most recent call last):
...
pyspark.errors.exceptions.base.UnsupportedOperationException: [PANDAS_API_ON_SPARK_FAIL_ON_ANSI_MODE] Pandas API on Spark does not properly work on ANSI mode.
Please set a Spark config 'spark.sql.ansi.enabled' to `false`.
Alternatively set a pandas-on-spark option 'compute.fail_on_ansi_mode' to `False` to force it to work, although it can cause unexpected behavior.
>>> ps.set_option('compute.fail_on_ansi_mode', False)
>>> ps.range(1)
...: PandasAPIOnSparkAdviceWarning: The config 'spark.sql.ansi.enabled' is set to True. This can cause unexpected behavior from pandas API on Spark since pandas API on Spark follows the behavior of pandas, not SQL.
warnings.warn(message, PandasAPIOnSparkAdviceWarning)
id
0 0
>>> spark.conf.set('spark.sql.ansi.enabled', False)
>>> ps.range(1)
id
0 0
```
### Why are the changes needed?
From Spark 4.0, ANSI mode is enabled by default, but pandas API on Spark won't work properly.
To avoid implicitly working wrongly, block it and ask the user how it should work.
### Does this PR introduce _any_ user-facing change?
Yes, pandas API on Spark will fail on ANSI mode by default.
### How was this patch tested?
Manually.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50816 from ueshin/issues/SPARK-52026/fail_on_ansi.
Authored-by: Takuya Ueshin <ue...@databricks.com>
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
python/docs/Makefile | 1 +
.../source/migration_guide/pyspark_upgrade.rst | 1 +
.../source/tutorial/pandas_on_spark/options.rst | 5 ++++
python/pyspark/errors/error-conditions.json | 7 +++++
python/pyspark/pandas/config.py | 20 ++++++++++----
.../pandas/tests/connect/test_parity_typedef.py | 3 ++-
.../pyspark/pandas/tests/plot/test_series_plot.py | 3 ++-
python/pyspark/pandas/tests/test_typedef.py | 3 ++-
python/pyspark/pandas/utils.py | 31 +++++++++++++---------
python/pyspark/testing/pandasutils.py | 1 -
10 files changed, 54 insertions(+), 21 deletions(-)
diff --git a/python/docs/Makefile b/python/docs/Makefile
index 045b03a1afd1..eba8a3fffbcd 100644
--- a/python/docs/Makefile
+++ b/python/docs/Makefile
@@ -22,6 +22,7 @@ SOURCEDIR ?= source
BUILDDIR ?= build
export PYTHONPATH=$(realpath ..):$(realpath ../lib/py4j-0.10.9.9-src.zip)
+export SPARK_ANSI_SQL_MODE=false
# Put it first so that "make" without argument is like "make help".
help:
diff --git a/python/docs/source/migration_guide/pyspark_upgrade.rst b/python/docs/source/migration_guide/pyspark_upgrade.rst
index 43caee536588..f91e29753964 100644
--- a/python/docs/source/migration_guide/pyspark_upgrade.rst
+++ b/python/docs/source/migration_guide/pyspark_upgrade.rst
@@ -83,6 +83,7 @@ Upgrading from PySpark 3.5 to 4.0
* In Spark 4.0, ``compute.ops_on_diff_frames`` is on by default. To restore the previous behavior, set ``compute.ops_on_diff_frames`` to ``false``.
* In Spark 4.0, the data type ``YearMonthIntervalType`` in ``DataFrame.collect`` no longer returns the underlying integers. To restore the previous behavior, set ``PYSPARK_YM_INTERVAL_LEGACY`` environment variable to ``1``.
* In Spark 4.0, items other than functions (e.g. ``DataFrame``, ``Column``, ``StructType``) have been removed from the wildcard import ``from pyspark.sql.functions import *``, you should import these items from proper modules (e.g. ``from pyspark.sql import DataFrame, Column``, ``from pyspark.sql.types import StructType``).
+* In Spark 4.0, pandas API on Spark will raise an exception if the underlying Spark is working with ANSI mode enabled that is enabled by default, as it will not work properly with ANSI mode. To make it work, you need to explicitly disable ANSI mode by setting ``spark.sql.ansi.enabled`` to ``false``. Alternatively you can set a pandas-on-spark option ``compute.fail_on_ansi_mode`` to ``False`` to force it to work, although it can cause unexpected behavior.
Upgrading from PySpark 3.3 to 3.4
diff --git a/python/docs/source/tutorial/pandas_on_spark/options.rst b/python/docs/source/tutorial/pandas_on_spark/options.rst
index cb3fbd8d427e..74197a5d9975 100644
--- a/python/docs/source/tutorial/pandas_on_spark/options.rst
+++ b/python/docs/source/tutorial/pandas_on_spark/options.rst
@@ -319,6 +319,11 @@ compute.isin_limit 80 'compute.isin_limit' set
better performance.
compute.pandas_fallback False 'compute.pandas_fallback' sets whether or not to
fallback automatically to Pandas' implementation.
+compute.fail_on_ansi_mode True 'compute.fail_on_ansi_mode' sets whether or not work
+ with ANSI mode. If True, pandas API on Spark raises
+ an exception if the underlying Spark is working with
+ ANSI mode enabled and the option
+ 'compute.ansi_mode_support' is False.
compute.ansi_mode_support False 'compute.ansi_mode_support' sets whether or not to
support the ANSI mode of the underlying Spark. If
False, pandas API on Spark may hit unexpected results
diff --git a/python/pyspark/errors/error-conditions.json b/python/pyspark/errors/error-conditions.json
index 9051ba666051..0b73f8482b7d 100644
--- a/python/pyspark/errors/error-conditions.json
+++ b/python/pyspark/errors/error-conditions.json
@@ -827,6 +827,13 @@
"<package_name> >= <minimum_version> must be installed; however, it was not found."
]
},
+ "PANDAS_API_ON_SPARK_FAIL_ON_ANSI_MODE": {
+ "message": [
+ "Pandas API on Spark does not properly work on ANSI mode.",
+ "Please set a Spark config 'spark.sql.ansi.enabled' to `false`.",
+ "Alternatively set a pandas-on-spark option 'compute.fail_on_ansi_mode' to `False` to force it to work, although it can cause unexpected behavior."
+ ]
+ },
"PANDAS_UDF_OUTPUT_EXCEEDS_INPUT_ROWS": {
"message": [
"The Pandas SCALAR_ITER UDF outputs more rows than input rows."
diff --git a/python/pyspark/pandas/config.py b/python/pyspark/pandas/config.py
index 2a10c022d05d..290bee4340c0 100644
--- a/python/pyspark/pandas/config.py
+++ b/python/pyspark/pandas/config.py
@@ -24,7 +24,7 @@ from typing import Any, Callable, Dict, Iterator, List, Tuple, Union, Optional
from pyspark._globals import _NoValue, _NoValueType
from pyspark.sql.session import SparkSession
-from pyspark.pandas.utils import default_session
+from pyspark.pandas.utils import default_session, is_testing
__all__ = ["get_option", "set_option", "reset_option", "options", "option_context"]
@@ -269,6 +269,16 @@ _options: List[Option] = [
default=False,
types=bool,
),
+ Option(
+ key="compute.fail_on_ansi_mode",
+ doc=(
+ "'compute.fail_on_ansi_mode' sets whether or not work with ANSI mode. "
+ "If True, pandas API on Spark raises an exception if the underlying Spark is "
+ "working with ANSI mode enabled and the option 'compute.ansi_mode_support' is False."
+ ),
+ default=True,
+ types=bool,
+ ),
Option(
key="compute.ansi_mode_support",
doc=(
@@ -277,7 +287,7 @@ _options: List[Option] = [
"If False, pandas API on Spark may hit unexpected results or errors. "
"The default is False."
),
- default=False,
+ default=is_testing(),
types=bool,
),
Option(
@@ -394,7 +404,7 @@ def get_option(
if default is _NoValue:
default = _options_dict[key].default
_options_dict[key].validate(default)
- spark_session = spark_session or default_session()
+ spark_session = spark_session or default_session(check_ansi_mode=False)
return json.loads(spark_session.conf.get(_key_format(key), default=json.dumps(default)))
@@ -419,7 +429,7 @@ def set_option(key: str, value: Any, *, spark_session: Optional[SparkSession] =
"""
_check_option(key)
_options_dict[key].validate(value)
- spark_session = spark_session or default_session()
+ spark_session = spark_session or default_session(check_ansi_mode=False)
spark_session.conf.set(_key_format(key), json.dumps(value))
@@ -443,7 +453,7 @@ def reset_option(key: str, *, spark_session: Optional[SparkSession] = None) -> N
None
"""
_check_option(key)
- spark_session = spark_session or default_session()
+ spark_session = spark_session or default_session(check_ansi_mode=False)
spark_session.conf.unset(_key_format(key))
diff --git a/python/pyspark/pandas/tests/connect/test_parity_typedef.py b/python/pyspark/pandas/tests/connect/test_parity_typedef.py
index 8df36ade9261..bfc556fd42d9 100644
--- a/python/pyspark/pandas/tests/connect/test_parity_typedef.py
+++ b/python/pyspark/pandas/tests/connect/test_parity_typedef.py
@@ -18,9 +18,10 @@ import unittest
from pyspark.pandas.tests.test_typedef import TypeHintTestsMixin
from pyspark.testing.connectutils import ReusedConnectTestCase
+from pyspark.testing.pandasutils import PandasOnSparkTestUtils
-class TypeHintParityTests(TypeHintTestsMixin, ReusedConnectTestCase):
+class TypeHintParityTests(TypeHintTestsMixin, PandasOnSparkTestUtils, ReusedConnectTestCase):
pass
diff --git a/python/pyspark/pandas/tests/plot/test_series_plot.py b/python/pyspark/pandas/tests/plot/test_series_plot.py
index 61d114f37b0e..35c9b443fd87 100644
--- a/python/pyspark/pandas/tests/plot/test_series_plot.py
+++ b/python/pyspark/pandas/tests/plot/test_series_plot.py
@@ -22,6 +22,7 @@ import numpy as np
from pyspark import pandas as ps
from pyspark.pandas.plot import PandasOnSparkPlotAccessor, BoxPlotBase
+from pyspark.testing.pandasutils import PandasOnSparkTestCase
from pyspark.testing.utils import have_plotly, plotly_requirement_message
@@ -83,7 +84,7 @@ class SeriesPlotTestsMixin:
self.assertEqual([50], result["fliers"])
-class SeriesPlotTests(SeriesPlotTestsMixin, unittest.TestCase):
+class SeriesPlotTests(SeriesPlotTestsMixin, PandasOnSparkTestCase):
pass
diff --git a/python/pyspark/pandas/tests/test_typedef.py b/python/pyspark/pandas/tests/test_typedef.py
index cb1917db2074..cac9aaf193a0 100644
--- a/python/pyspark/pandas/tests/test_typedef.py
+++ b/python/pyspark/pandas/tests/test_typedef.py
@@ -54,6 +54,7 @@ from pyspark.pandas.typedef import (
pandas_on_spark_type,
)
from pyspark import pandas as ps
+from pyspark.testing.pandasutils import PandasOnSparkTestCase
class TypeHintTestsMixin:
@@ -431,7 +432,7 @@ class TypeHintTestsMixin:
self.assertEqual(pandas_on_spark_type(extension_dtype), (extension_dtype, spark_type))
-class TypeHintTests(TypeHintTestsMixin, unittest.TestCase):
+class TypeHintTests(TypeHintTestsMixin, PandasOnSparkTestCase):
pass
diff --git a/python/pyspark/pandas/utils.py b/python/pyspark/pandas/utils.py
index f662e2dfb3ce..f4528ed63cf7 100644
--- a/python/pyspark/pandas/utils.py
+++ b/python/pyspark/pandas/utils.py
@@ -43,7 +43,7 @@ from pandas.api.types import is_list_like # type: ignore[attr-defined]
from pyspark.sql import functions as F, Column, DataFrame as PySparkDataFrame, SparkSession
from pyspark.sql.types import DoubleType
from pyspark.sql.utils import is_remote
-from pyspark.errors import PySparkTypeError
+from pyspark.errors import PySparkTypeError, UnsupportedOperationException
from pyspark import pandas as ps # noqa: F401
from pyspark.pandas._typing import (
Axis,
@@ -476,21 +476,28 @@ def is_testing() -> bool:
return "SPARK_TESTING" in os.environ
-def default_session() -> SparkSession:
+def default_session(*, check_ansi_mode: bool = True) -> SparkSession:
spark = SparkSession.getActiveSession()
if spark is None:
spark = SparkSession.builder.appName("pandas-on-Spark").getOrCreate()
- if (
- not ps.get_option("compute.ansi_mode_support", spark_session=spark)
- and spark.conf.get("spark.sql.ansi.enabled") == "true"
- ):
- log_advice(
- "The config 'spark.sql.ansi.enabled' is set to True. "
- "This can cause unexpected behavior "
- "from pandas API on Spark since pandas API on Spark follows "
- "the behavior of pandas, not SQL."
- )
+ if check_ansi_mode:
+ if (
+ not ps.get_option("compute.ansi_mode_support", spark_session=spark)
+ and spark.conf.get("spark.sql.ansi.enabled") == "true"
+ ):
+ if ps.get_option("compute.fail_on_ansi_mode", spark_session=spark):
+ raise UnsupportedOperationException(
+ errorClass="PANDAS_API_ON_SPARK_FAIL_ON_ANSI_MODE",
+ messageParameters={},
+ )
+ else:
+ log_advice(
+ "The config 'spark.sql.ansi.enabled' is set to True. "
+ "This can cause unexpected behavior "
+ "from pandas API on Spark since pandas API on Spark follows "
+ "the behavior of pandas, not SQL."
+ )
return spark
diff --git a/python/pyspark/testing/pandasutils.py b/python/pyspark/testing/pandasutils.py
index 028a058cc601..730212f5baa2 100644
--- a/python/pyspark/testing/pandasutils.py
+++ b/python/pyspark/testing/pandasutils.py
@@ -475,7 +475,6 @@ class PandasOnSparkTestCase(ReusedSQLTestCase, PandasOnSparkTestUtils):
def setUpClass(cls):
super(PandasOnSparkTestCase, cls).setUpClass()
cls.spark.conf.set(SPARK_CONF_ARROW_ENABLED, True)
- ps.set_option("compute.ansi_mode_support", True, spark_session=cls.spark)
def setUp(self):
super().setUp()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org