[jira] [Created] (SPARK-45988) Fix `pyspark.pandas.tests.computation.test_apply_func` in Python 3.11

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

Dongjoon Hyun created SPARK-45988:
-------------------------------------

             Summary: Fix `pyspark.pandas.tests.computation.test_apply_func` in Python 3.11
                 Key: SPARK-45988
                 URL: https://issues.apache.org/jira/browse/SPARK-45988
             Project: Spark
          Issue Type: Sub-task
          Components: PySpark
    Affects Versions: 4.0.0
            Reporter: Dongjoon Hyun


https://github.com/apache/spark/actions/runs/6914662405/job/18812759697

{code}
======================================================================
ERROR [0.686s]: test_apply_batch_with_type (pyspark.pandas.tests.computation.test_apply_func.FrameApplyFunctionTests.test_apply_batch_with_type)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/pandas/tests/computation/test_apply_func.py", line 248, in test_apply_batch_with_type
    def identify3(x) -> ps.DataFrame[float, [int, List[int]]]:
                        ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/frame.py", line 13540, in __class_getitem__
    return create_tuple_for_frame_type(params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 721, in create_tuple_for_frame_type
    return Tuple[_to_type_holders(params)]
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 766, in _to_type_holders
    data_types = _new_type_holders(data_types, NameTypeHolder)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/__w/spark/spark/python/pyspark/pandas/typedef/typehints.py", line 832, in _new_type_holders
    raise TypeError(
TypeError: Type hints should be specified as one of:
  - DataFrame[type, type, ...]
  - DataFrame[name: type, name: type, ...]
  - DataFrame[dtypes instance]
  - DataFrame[zip(names, types)]
  - DataFrame[index_type, [type, ...]]
  - DataFrame[(index_name, index_type), [(name, type), ...]]
  - DataFrame[dtype instance, dtypes instance]
  - DataFrame[(index_name, index_type), zip(names, types)]
  - DataFrame[[index_type, ...], [type, ...]]
  - DataFrame[[(index_name, index_type), ...], [(name, type), ...]]
  - DataFrame[dtypes instance, dtypes instance]
  - DataFrame[zip(index_names, index_types), zip(names, types)]
However, got (<class 'int'>, typing.List[int]).

----------------------------------------------------------------------
Ran 10 tests in 34.327s

FAILED (errors=1)
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org