You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/04/16 04:54:00 UTC
[jira] [Updated] (SPARK-51814) Introduce a new API `transformWithState` in PySpark
[ https://issues.apache.org/jira/browse/SPARK-51814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-51814:
-----------------------------------
Labels: pull-request-available (was: )
> Introduce a new API `transformWithState` in PySpark
> ---------------------------------------------------
>
> Key: SPARK-51814
> URL: https://issues.apache.org/jira/browse/SPARK-51814
> Project: Spark
> Issue Type: New Feature
> Components: Structured Streaming
> Affects Versions: 4.1.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> We introduced a new API named `transformWithState` in Scala and PySpark in Spark 4.0.0.
> In PySpark, we use Pandas as a type of input and return type for user function, since it has been considered as de-facto standard library, and majority UDF APIs in PySpark support Pandas.
> While this helps performing vectorized operations in user function, we realized the overhead on converting Arrow RecordBatch to Pandas DataFrame is non-trivial, especially the case where cardinality is quite high (e.g. every data has its unique key). The issue is mostly specific to streaming.
> As a resolution of the issue, we are proposing a new API named `transformWithState` (same with Scala) in PySpark. This would expose PySpark Row type as a type of input and return type for user function - we verified the overhead on conversion is quite smaller than Pandas. Also, PySpark Row type fits to the type when users use struct type of column as an input for normal PySpark UDF.
> (We considered Arrow as a type since there are several pairs of Arrow <-> Pandas for existing PySpark API. We rejected the approach because we don't see a strong evidence that Arrow type of API is being widely used, and PySpark Row is lot easier to use.)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org