You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2025/03/05 02:13:50 UTC
(spark) 01/02: Revert "[SPARK-48516][PYTHON][FOLLOW-UP] Add a note in migration guide about Arrow-optimized Python UDF enabled by default"
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
commit 71c91cc76b9551ab9937c1d5f58f057037619bdb
Author: Hyukjin Kwon <gu...@apache.org>
AuthorDate: Wed Mar 5 11:11:06 2025 +0900
Revert "[SPARK-48516][PYTHON][FOLLOW-UP] Add a note in migration guide about Arrow-optimized Python UDF enabled by default"
This reverts commit 45900c434c9ae275e9394a6fa0936a34a8bf3f3b.
---
python/docs/source/migration_guide/pyspark_upgrade.rst | 1 -
1 file changed, 1 deletion(-)
diff --git a/python/docs/source/migration_guide/pyspark_upgrade.rst b/python/docs/source/migration_guide/pyspark_upgrade.rst
index 8ef66a79cd70..906e40140cef 100644
--- a/python/docs/source/migration_guide/pyspark_upgrade.rst
+++ b/python/docs/source/migration_guide/pyspark_upgrade.rst
@@ -75,7 +75,6 @@ Upgrading from PySpark 3.5 to 4.0
* In Spark 4.0, ``compute.ops_on_diff_frames`` is on by default. To restore the previous behavior, set ``compute.ops_on_diff_frames`` to ``false``.
* In Spark 4.0, the data type ``YearMonthIntervalType`` in ``DataFrame.collect`` no longer returns the underlying integers. To restore the previous behavior, set ``PYSPARK_YM_INTERVAL_LEGACY`` environment variable to ``1``.
* In Spark 4.0, items other than functions (e.g. ``DataFrame``, ``Column``, ``StructType``) have been removed from the wildcard import ``from pyspark.sql.functions import *``, you should import these items from proper modules (e.g. ``from pyspark.sql import DataFrame, Column``, ``from pyspark.sql.types import StructType``).
-* In Spark 4.0, ``spark.sql.execution.pythonUDF.arrow.enabled`` is enabled by default. If users have PyArrow and pandas installed in their local and Spark Cluster, it automatically optimizes the regular Python UDFs with Arrow. To turn off the Arrow optimization, set ``spark.sql.execution.pythonUDF.arrow.enabled`` to ``false``.
Upgrading from PySpark 3.3 to 3.4
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org