You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yicong Huang (Jira)" <ji...@apache.org> on 2026/04/04 05:26:00 UTC
[jira] [Updated] (SPARK-56351) Add dedicated documentation page for Arrow Python UDFs
[ https://issues.apache.org/jira/browse/SPARK-56351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yicong Huang updated SPARK-56351:
---------------------------------
Description: Currently, the Arrow Python UDF documentation is a small section (about 30 lines) embedded within the Pandas UDF documentation page (arrow_pandas.rst). Given that Arrow Python UDFs are a distinct feature (regular Python UDFs using Arrow for serialization, enabled via useArrow=True or spark.sql.execution.pythonUDF.arrow.enabled), they deserve a dedicated documentation page. (was: Currently, the Arrow Python UDF documentation is a small section (about 30 lines) embedded within the Pandas UDF documentation page (arrow_pandas.rst). Given that Arrow Python UDFs are a distinct feature (regular Python UDFs using Arrow for serialization, enabled via useArrow=True or spark.sql.execution.pythonUDF.arrow.enabled), they deserve a dedicated documentation page.
Proposed changes:
1. Create a new arrow_python_udf.rst page with comprehensive documentation for Arrow Python UDFs, including usage examples, type coercion details, and configuration options.
2. Move the existing Arrow Python UDFs section from arrow_pandas.rst to the new page, replacing it with a cross-reference link.
3. Add the new page to the tutorial/sql/index.rst table of contents.)
> Add dedicated documentation page for Arrow Python UDFs
> ------------------------------------------------------
>
> Key: SPARK-56351
> URL: https://issues.apache.org/jira/browse/SPARK-56351
> Project: Spark
> Issue Type: Improvement
> Components: Documentation, PySpark
> Affects Versions: 4.2.0
> Reporter: Yicong Huang
> Priority: Major
>
> Currently, the Arrow Python UDF documentation is a small section (about 30 lines) embedded within the Pandas UDF documentation page (arrow_pandas.rst). Given that Arrow Python UDFs are a distinct feature (regular Python UDFs using Arrow for serialization, enabled via useArrow=True or spark.sql.execution.pythonUDF.arrow.enabled), they deserve a dedicated documentation page.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org