[jira] [Comment Edited] (ARROW-3850) [Python] Support MapType and StructType for enhanced PySpark integration

["Bryan Cutler (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17236400#comment-17236400 ] 

Bryan Cutler edited comment on ARROW-3850 at 11/20/20, 7:47 PM:
----------------------------------------------------------------

MapType has been added to PySpark when using PyArrow 2.0.0+, I think we can close this now.


was (Author: bryanc):
MapType has been added to PySpark, I think we can close this now.

> [Python] Support MapType and StructType for enhanced PySpark integration
> ------------------------------------------------------------------------
>
>                 Key: ARROW-3850
>                 URL: https://issues.apache.org/jira/browse/ARROW-3850
>             Project: Apache Arrow
>          Issue Type: New Feature
>          Components: Python
>    Affects Versions: 0.11.1
>            Reporter: Florian Wilhelm
>            Priority: Major
>             Fix For: 3.0.0
>
>
> It would be great to support MapType and (nested) StructType in Arrow so that PySpark can make use of it.
>  
>  Quite often as in my use-case in Hive table cells are also complex types saved. Currently it's not possible to user the new {{[pandas_udf|https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=explode#pyspark.sql.functions.pandas_udf]}} decorator which internally uses Arrow to generate a UDF for columns with complex types.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)