[GitHub] [spark] BryanCutler opened a new pull request #24306: [SPARK-27387][PYTHON][TESTS] Replace sqlutils.assertPandasEqual with Pandas assert_frame_equals

[GitBox <gi...@apache.org>]

BryanCutler opened a new pull request #24306: [SPARK-27387][PYTHON][TESTS] Replace sqlutils.assertPandasEqual with Pandas assert_frame_equals 
URL: https://github.com/apache/spark/pull/24306
 
 
   ## What changes were proposed in this pull request?
   
   Running PySpark tests with Pandas 0.24.x causes a failure in `test_pandas_udf_grouped_map` test_supported_types:
   
   ```
   ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
   ```
   
   This is because a column is an ArrayType and the method `sqlutils ReusedSQLTestCase.assertPandasEqual ` does not properly check this.
   
   This PR removes `assertPandasEqual` and replaces it with the built-in `pandas.util.testing.assert_frame_equal` which can properly handle columns of ArrayType and also prints out better diff between the DataFrames when an error occurs.
   
   Additionally, imports of pandas and pyarrow were moved to the top of related test files to avoid duplicating the same import many times.
   
   ## How was this patch tested?
   
   Existing tests
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org