[jira] [Created] (SPARK-37885) Allow pandas_udf to take type annotations with future annotations enabled

["Takuya Ueshin (Jira)" <ji...@apache.org>]

Takuya Ueshin created SPARK-37885:
-------------------------------------

             Summary: Allow pandas_udf to take type annotations with future annotations enabled
                 Key: SPARK-37885
                 URL: https://issues.apache.org/jira/browse/SPARK-37885
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 3.3.0
            Reporter: Takuya Ueshin


When using {{{}from __future__ import annotations{}}}, the type hints will be all strings, then pandas UDF type inference won't work as follows:
{code:python}
>>> from __future__ import annotations
>>> from typing import Union
>>> import pandas as pd
>>> from pyspark.sql.functions import pandas_udf
>>> @pandas_udf("long")
... def plus_one(v: Union[pd.Series, pd.DataFrame]) -> pd.Series:
...     return v + 1

Traceback (most recent call last):
...
NotImplementedError: Unsupported signature: (v: 'Union[pd.Series, pd.DataFrame]') -> 'pd.Series'.
{code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org