You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Wes McKinney (Jira)" <ji...@apache.org> on 2020/06/26 16:00:00 UTC
[jira] [Commented] (ARROW-3850) [Python] Support MapType and
StructType for enhanced PySpark integration
[ https://issues.apache.org/jira/browse/ARROW-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146432#comment-17146432 ]
Wes McKinney commented on ARROW-3850:
-------------------------------------
ping [~bryanc]
> [Python] Support MapType and StructType for enhanced PySpark integration
> ------------------------------------------------------------------------
>
> Key: ARROW-3850
> URL: https://issues.apache.org/jira/browse/ARROW-3850
> Project: Apache Arrow
> Issue Type: New Feature
> Components: Python
> Affects Versions: 0.11.1
> Reporter: Florian Wilhelm
> Priority: Major
> Fix For: 1.0.0
>
>
> It would be great to support MapType and (nested) StructType in Arrow so that PySpark can make use of it.
>
> Quite often as in my use-case in Hive table cells are also complex types saved. Currently it's not possible to user the new {{[pandas_udf|https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=explode#pyspark.sql.functions.pandas_udf]}} decorator which internally uses Arrow to generate a UDF for columns with complex types.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)