[jira] [Commented] (SPARK-34693) Support null in conversions to and from Arrow

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-34693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17301376#comment-17301376 ] 

Hyukjin Kwon commented on SPARK-34693:
--------------------------------------

Which codes did you run? It would be great if you can provide the reproducer

> Support null in conversions to and from Arrow
> ---------------------------------------------
>
>                 Key: SPARK-34693
>                 URL: https://issues.apache.org/jira/browse/SPARK-34693
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 3.1.1
>            Reporter: Laurens
>            Priority: Minor
>
> Looks like a regression from or related to SPARK-33489.
> I got the following running spark 3.1.1. with koalas 1.7.0
> {code:java}
> TypeError                                 Traceback (most recent call last)
> /var/scratch/miniconda3/lib/python3.8/site-packages/pyspark/sql/udf.py in returnType(self)
>     100             try:
> --> 101                 to_arrow_type(self._returnType_placeholder)
>     102             except TypeError:
> /var/scratch/miniconda3/lib/python3.8/site-packages/pyspark/sql/pandas/types.py in to_arrow_type(dt)
>      75     else:
> ---> 76         raise TypeError("Unsupported type in conversion to Arrow: " + str(dt))
>      77     return arrow_type
> TypeError: Unsupported type in conversion to Arrow: NullType
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org