[GitHub] [spark] xinrong-meng commented on a diff in pull request #42568: [SPARK-44876][PYTHON] Enable and fix test_parity_arrow_python_udf.

["xinrong-meng (via GitHub)" <gi...@apache.org>]

xinrong-meng commented on code in PR #42568:
URL: https://github.com/apache/spark/pull/42568#discussion_r1298871550


##########
python/pyspark/sql/udf.py:
##########
@@ -131,58 +130,24 @@ def _create_py_udf(
     else:
         is_arrow_enabled = useArrow
 
-    regular_udf = _create_udf(f, returnType, PythonEvalType.SQL_BATCHED_UDF)
-    try:
-        is_func_with_args = len(getfullargspec(f).args) > 0
-    except TypeError:
-        is_func_with_args = False
+    eval_type: int = PythonEvalType.SQL_BATCHED_UDF
+
     if is_arrow_enabled:
+        try:
+            is_func_with_args = len(getfullargspec(f).args) > 0
+        except TypeError:
+            is_func_with_args = False
         if is_func_with_args:
-            return _create_arrow_py_udf(regular_udf)
+            require_minimum_pandas_version()
+            require_minimum_pyarrow_version()

Review Comment:
   I would suggest for Arrow Python UDF, we fail in Spark 3.5. And when it's turned on by default in Spark 4.0, we fall back. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org