You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/07/03 08:02:00 UTC
[jira] [Updated] (SPARK-28240) Fix Arrow tests to pass with Python
2.7 and latest PyArrow and Pandas in PySpark
[ https://issues.apache.org/jira/browse/SPARK-28240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-28240:
---------------------------------
Summary: Fix Arrow tests to pass with Python 2.7 and latest PyArrow and Pandas in PySpark (was: Fix Arrow tests to pass with Python 2.7, PyArrow 0.13, Pandas 0.24.2 in PySpark)
> Fix Arrow tests to pass with Python 2.7 and latest PyArrow and Pandas in PySpark
> --------------------------------------------------------------------------------
>
> Key: SPARK-28240
> URL: https://issues.apache.org/jira/browse/SPARK-28240
> Project: Spark
> Issue Type: Test
> Components: PySpark
> Affects Versions: 3.0.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
> {code}
> ======================================================================
> FAIL: test_createDataFrame_with_incorrect_schema (pyspark.sql.tests.test_arrow.ArrowTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/Users/hyukjin.kwon/workspace/forked/spark/python/pyspark/sql/tests/test_arrow.py", line 275, in test_createDataFrame_with_incorrect_schema
> self.spark.createDataFrame(pdf, schema=wrong_schema)
> AssertionError: "integer.*required.*got.*str" does not match "('Exception thrown when converting pandas.Series (object) to Arrow Array (int32). It can be caused by overflows or other unsafe conversions warned by Arrow. Arrow safe type check can be disabled by using SQL config `spark.sql.execution.pandas.arrowSafeTypeConversion`.', ArrowTypeError('an integer is required',))"
> ======================================================================
> FAIL: test_createDataFrame_with_incorrect_schema (pyspark.sql.tests.test_arrow.EncryptionArrowTests)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/Users/hyukjin.kwon/workspace/forked/spark/python/pyspark/sql/tests/test_arrow.py", line 275, in test_createDataFrame_with_incorrect_schema
> self.spark.createDataFrame(pdf, schema=wrong_schema)
> AssertionError: "integer.*required.*got.*str" does not match "('Exception thrown when converting pandas.Series (object) to Arrow Array (int32). It can be caused by overflows or other unsafe conversions warned by Arrow. Arrow safe type check can be disabled by using SQL config `spark.sql.execution.pandas.arrowSafeTypeConversion`.', ArrowTypeError('an integer is required',))"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org