[GitHub] [spark] HyukjinKwon commented on pull request #35391: [SPARK-38098][PYTHON] Add support for ArrayType of nested StructType to arrow-based conversion

[GitBox <gi...@apache.org>]

HyukjinKwon commented on pull request #35391:
URL: https://github.com/apache/spark/pull/35391#issuecomment-1075922395


   @LucaCanali thanks for addressing my comment. The test seems like failing:
   
   ```
   
   ======================================================================
   ERROR [0.182s]: test_pandas_array_struct (pyspark.sql.tests.test_pandas_udf_scalar.ScalarPandasUDFTests)
   ----------------------------------------------------------------------
   Traceback (most recent call last):
     File "/__w/spark/spark/python/pyspark/sql/tests/test_pandas_udf_scalar.py", line 152, in test_pandas_array_struct
       result = df.select(return_cols("array_struct_col"))
     File "/__w/spark/spark/python/pyspark/sql/udf.py", line 276, in wrapper
       return self(*args)
     File "/__w/spark/spark/python/pyspark/sql/udf.py", line 249, in __call__
       judf = self._judf
     File "/__w/spark/spark/python/pyspark/sql/udf.py", line 215, in _judf
       self._judf_placeholder = self._create_judf(self.func)
     File "/__w/spark/spark/python/pyspark/sql/udf.py", line 224, in _create_judf
       wrapped_func = _wrap_function(sc, func, self.returnType)
     File "/__w/spark/spark/python/pyspark/sql/udf.py", line 50, in _wrap_function
       pickled_command, broadcast_vars, env, includes = _prepare_for_python_RDD(sc, command)
     File "/__w/spark/spark/python/pyspark/rdd.py", line 3350, in _prepare_for_python_RDD
       pickled_command = ser.dumps(command)
     File "/__w/spark/spark/python/pyspark/serializers.py", line 458, in dumps
       return cloudpickle.dumps(obj, pickle_protocol)
     File "/__w/spark/spark/python/pyspark/cloudpickle/cloudpickle_fast.py", line 73, in dumps
       cp.dump(obj)
     File "/__w/spark/spark/python/pyspark/cloudpickle/cloudpickle_fast.py", line 602, in dump
       return Pickler.dump(self, obj)
     File "/__w/spark/spark/python/pyspark/cloudpickle/cloudpickle_fast.py", line 316, in _file_reduce
       raise pickle.PicklingError(
   _pickle.PicklingError: Cannot pickle files that are not opened for reading: w
   ```
   
   Could we fix this one?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org