[jira] [Commented] (SPARK-31374) Returning complex types in Pandas UDF

["Oleksii Kachaiev (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-31374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098609#comment-17098609 ] 

Oleksii Kachaiev commented on SPARK-31374:
------------------------------------------

[~hoeze] do you have an example of the code of {{spark_window_overlap}} function?

> Returning complex types in Pandas UDF
> -------------------------------------
>
>                 Key: SPARK-31374
>                 URL: https://issues.apache.org/jira/browse/SPARK-31374
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 3.0.0
>            Reporter: F. H.
>            Priority: Major
>              Labels: features
>
> I would like to return a complex type in an GROUPED_AGG operation:
> {code:python}
> window_overlap_schema = t.StructType([
>      t.StructField("counts", t.ArrayType(t.LongType())),
>      t.StructField("starts", t.ArrayType(t.LongType())),
>      t.StructField("ends", t.ArrayType(t.LongType())),
> ])
> @f.pandas_udf(window_overlap_schema, f.PandasUDFType.GROUPED_AGG)
> def spark_window_overlap([...]):
>     [...]
> {code}
> However, I get the following error when trying to run this:
> {code:python}
> NotImplementedError: Invalid returnType with grouped aggregate Pandas UDFs: StructType(List(StructField(counts,ArrayType(LongType,true),true),StructField(starts,ArrayType(LongType,true),true),StructField(ends,ArrayType(LongType,true),true))) is not supported
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org