You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Aki Ariga (Jira)" <ji...@apache.org> on 2020/03/04 05:44:00 UTC
[jira] [Commented] (SPARK-30966) spark.createDataFrame fails with
pandas DataFrame including pandas.NA
[ https://issues.apache.org/jira/browse/SPARK-30966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17050906#comment-17050906 ]
Aki Ariga commented on SPARK-30966:
-----------------------------------
This should be blocked by https://issues.apache.org/jira/browse/ARROW-5379
> spark.createDataFrame fails with pandas DataFrame including pandas.NA
> ----------------------------------------------------------------------
>
> Key: SPARK-30966
> URL: https://issues.apache.org/jira/browse/SPARK-30966
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.4.5
> Reporter: Aki Ariga
> Priority: Major
>
> As of pandas 1.0.0, pandas.NA was introduced, and that breaks createDataFrame function as the following:
> {code:python}
> In [5]: from pyspark.sql import SparkSession
> In [6]: spark = SparkSession.builder.getOrCreate()
> In [7]: spark.conf.set("spark.sql.execution.arrow.enabled", "true")
> In [8]: import numpy as np
> ...: import pandas as pd
> In [12]: pdf = pd.DataFrame(data=[{'a':1,'b':2}, {'a':3,'b':4,'c':5}], dtype=pd.Int64Dtype())
> In [16]: pdf
> Out[16]:
> a b c
> 0 1 2 <NA>
> 1 3 4 5
> In [13]: sdf = spark.createDataFrame(pdf)
> /Users/ariga/src/pytd/.venv/lib/python3.6/site-packages/pyspark/sql/session.py:714: UserWarning: createDataFrame attempted Arrow optimization because 'spark.sql.execution.arrow.enabled' is set to true; however, failed by the reason below:
> Did not pass numpy.dtype object
> Attempting non-optimization as 'spark.sql.execution.arrow.fallback.enabled' is set to true.
> warnings.warn(msg)
> ---------------------------------------------------------------------------
> TypeError Traceback (most recent call last)
> <ipython-input-13-ad13ba53b87e> in <module>
> ----> 1 sdf = spark.createDataFrame(df2)
> ~/src/pytd/.venv/lib/python3.6/site-packages/pyspark/sql/session.py in createDataFrame(self, data, schema, samplingRatio, verifySchema)
> 746 rdd, schema = self._createFromRDD(data.map(prepare), schema, samplingRatio)
> 747 else:
> --> 748 rdd, schema = self._createFromLocal(map(prepare, data), schema)
> 749 jrdd = self._jvm.SerDeUtil.toJavaArray(rdd._to_java_object_rdd())
> 750 jdf = self._jsparkSession.applySchemaToPythonRDD(jrdd.rdd(), schema.json())
> ~/src/pytd/.venv/lib/python3.6/site-packages/pyspark/sql/session.py in _createFromLocal(self, data, schema)
> 414
> 415 if schema is None or isinstance(schema, (list, tuple)):
> --> 416 struct = self._inferSchemaFromList(data, names=schema)
> 417 converter = _create_converter(struct)
> 418 data = map(converter, data)
> ~/src/pytd/.venv/lib/python3.6/site-packages/pyspark/sql/session.py in _inferSchemaFromList(self, data, names)
> 346 warnings.warn("inferring schema from dict is deprecated,"
> 347 "please use pyspark.sql.Row instead")
> --> 348 schema = reduce(_merge_type, (_infer_schema(row, names) for row in data))
> 349 if _has_nulltype(schema):
> 350 raise ValueError("Some of types cannot be determined after inferring")
> ~/src/pytd/.venv/lib/python3.6/site-packages/pyspark/sql/types.py in _merge_type(a, b, name)
> 1099 fields = [StructField(f.name, _merge_type(f.dataType, nfs.get(f.name, NullType()),
> 1100 name=new_name(f.name)))
> -> 1101 for f in a.fields]
> 1102 names = set([f.name for f in fields])
> 1103 for n in nfs:
> ~/src/pytd/.venv/lib/python3.6/site-packages/pyspark/sql/types.py in <listcomp>(.0)
> 1099 fields = [StructField(f.name, _merge_type(f.dataType, nfs.get(f.name, NullType()),
> 1100 name=new_name(f.name)))
> -> 1101 for f in a.fields]
> 1102 names = set([f.name for f in fields])
> 1103 for n in nfs:
> ~/src/pytd/.venv/lib/python3.6/site-packages/pyspark/sql/types.py in _merge_type(a, b, name)
> 1092 elif type(a) is not type(b):
> 1093 # TODO: type cast (such as int -> long)
> -> 1094 raise TypeError(new_msg("Can not merge type %s and %s" % (type(a), type(b))))
> 1095
> 1096 # same type
> TypeError: field c: Can not merge type <class 'pyspark.sql.types.StructType'> and <class 'pyspark.sql.types.LongType'>
> In [15]: pyspark.__version__
> Out[15]: '2.4.5'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org