You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2025/02/22 00:23:16 UTC
(spark) branch master updated: [SPARK-51276][PYTHON] Enable spark.sql.execution.arrow.pyspark.enabled by default
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 9ac566dea896 [SPARK-51276][PYTHON] Enable spark.sql.execution.arrow.pyspark.enabled by default
9ac566dea896 is described below
commit 9ac566dea89644b6054bf1362b9647e1dc67d3af
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Sat Feb 22 09:22:53 2025 +0900
[SPARK-51276][PYTHON] Enable spark.sql.execution.arrow.pyspark.enabled by default
### What changes were proposed in this pull request?
This PR proposes to enable `spark.sql.execution.arrow.pyspark.enabled` by default.
### Why are the changes needed?
So the end users can leverage the optimization by default.
### Does this PR introduce _any_ user-facing change?
It will fallback to non-optimized code path so it impact will be minimized. Users will leverage Arrow optimization by default.
### How was this patch tested?
Existing tests in the CI.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #50036 from HyukjinKwon/enable-arrow-bydefault.
Authored-by: Hyukjin Kwon <gu...@apache.org>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/docs/source/migration_guide/pyspark_upgrade.rst | 1 +
python/pyspark/pandas/base.py | 5 +++--
python/pyspark/sql/tests/connect/test_connect_creation.py | 5 +++++
.../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 2 +-
4 files changed, 10 insertions(+), 3 deletions(-)
diff --git a/python/docs/source/migration_guide/pyspark_upgrade.rst b/python/docs/source/migration_guide/pyspark_upgrade.rst
index 8ef66a79cd70..976aef7cb68b 100644
--- a/python/docs/source/migration_guide/pyspark_upgrade.rst
+++ b/python/docs/source/migration_guide/pyspark_upgrade.rst
@@ -76,6 +76,7 @@ Upgrading from PySpark 3.5 to 4.0
* In Spark 4.0, the data type ``YearMonthIntervalType`` in ``DataFrame.collect`` no longer returns the underlying integers. To restore the previous behavior, set ``PYSPARK_YM_INTERVAL_LEGACY`` environment variable to ``1``.
* In Spark 4.0, items other than functions (e.g. ``DataFrame``, ``Column``, ``StructType``) have been removed from the wildcard import ``from pyspark.sql.functions import *``, you should import these items from proper modules (e.g. ``from pyspark.sql import DataFrame, Column``, ``from pyspark.sql.types import StructType``).
* In Spark 4.0, ``spark.sql.execution.pythonUDF.arrow.enabled`` is enabled by default. If users have PyArrow and pandas installed in their local and Spark Cluster, it automatically optimizes the regular Python UDFs with Arrow. To turn off the Arrow optimization, set ``spark.sql.execution.pythonUDF.arrow.enabled`` to ``false``.
+* In Spark 4.0, ``spark.sql.execution.arrow.pyspark.enabled`` is enabled by default. If users have PyArrow and pandas installed in their local and Spark Cluster, it automatically makes use of Apache Arrow for columnar data transfers in PySpark. This optimization applies to ``pyspark.sql.DataFrame.toPandas`` and ``pyspark.sql.SparkSession.createDataFrame`` when its input is a Pandas DataFrame or a NumPy ndarray. To turn off the Arrow optimization, set ``spark.sql.execution.arrow.pyspark.e [...]
Upgrading from PySpark 3.3 to 3.4
diff --git a/python/pyspark/pandas/base.py b/python/pyspark/pandas/base.py
index 01e23214d662..81358185a560 100644
--- a/python/pyspark/pandas/base.py
+++ b/python/pyspark/pandas/base.py
@@ -1191,6 +1191,7 @@ class IndexOpsMixin(object, metaclass=ABCMeta):
return self._with_new_scol(col, field=self._internal.data_fields[0].copy(nullable=True))
# TODO: Update Documentation for Bins Parameter when its supported
+ # TODO(SPARK-51287): Enable s.index.value_counts() tests
def value_counts(
self,
normalize: bool = False,
@@ -1323,7 +1324,7 @@ class IndexOpsMixin(object, metaclass=ABCMeta):
('falcon', 'length')],
)
- >>> s.index.value_counts().sort_index()
+ >>> s.index.value_counts().sort_index() # doctest: +SKIP
(cow, length) 1
(cow, weight) 2
(falcon, length) 2
@@ -1331,7 +1332,7 @@ class IndexOpsMixin(object, metaclass=ABCMeta):
(lama, weight) 3
Name: count, dtype: int64
- >>> s.index.value_counts(normalize=True).sort_index()
+ >>> s.index.value_counts(normalize=True).sort_index() # doctest: +SKIP
(cow, length) 0.111111
(cow, weight) 0.222222
(falcon, length) 0.222222
diff --git a/python/pyspark/sql/tests/connect/test_connect_creation.py b/python/pyspark/sql/tests/connect/test_connect_creation.py
index 5352913f6609..163a1c17bfaf 100644
--- a/python/pyspark/sql/tests/connect/test_connect_creation.py
+++ b/python/pyspark/sql/tests/connect/test_connect_creation.py
@@ -219,6 +219,11 @@ class SparkConnectCreationTests(SparkConnectSQLTestCase):
self.assert_eq(sdf.toPandas(), cdf.toPandas())
def test_with_none_and_nan(self):
+ # TODO(SPARK-51286): Fix test_with_none_and_nan to to pass with Arrow enabled
+ with self.sql_conf({"spark.sql.execution.arrow.pyspark.enabled": False}):
+ self.check_with_none_and_nan()
+
+ def check_with_none_and_nan(self):
# SPARK-41855: make createDataFrame support None and NaN
# SPARK-41814: test with eqNullSafe
data1 = [Row(id=1, value=float("NaN")), Row(id=2, value=42.0), Row(id=3, value=None)]
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 5b42c157e598..d5e641287385 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -3271,7 +3271,7 @@ object SQLConf {
.doc("(Deprecated since Spark 3.0, please set 'spark.sql.execution.arrow.pyspark.enabled'.)")
.version("2.3.0")
.booleanConf
- .createWithDefault(false)
+ .createWithDefault(true)
val ARROW_PYSPARK_EXECUTION_ENABLED =
buildConf("spark.sql.execution.arrow.pyspark.enabled")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org