You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ruslan Dautkhanov (JIRA)" <ji...@apache.org> on 2017/11/13 07:59:00 UTC
[jira] [Created] (SPARK-22505) toDF() / createDataFrame() type
inference doesn't work as expected
Ruslan Dautkhanov created SPARK-22505:
-----------------------------------------
Summary: toDF() / createDataFrame() type inference doesn't work as expected
Key: SPARK-22505
URL: https://issues.apache.org/jira/browse/SPARK-22505
Project: Spark
Issue Type: Bug
Components: PySpark, Spark Core
Affects Versions: 2.2.0
Reporter: Ruslan Dautkhanov
{code}
df = sc.parallelize([('1','a'),('2','b'),('3','c')]).toDF(['should_be_int','should_be_str'])
df.printSchema()
{code}
produces
{noformat}
root
|-- should_be_int: string (nullable = true)
|-- should_be_str: string (nullable = true)
{noformat}
Notice `should_be_int` has `string` datatype, according to documentation:
https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection
{quote}
Spark SQL can convert an RDD of Row objects to a DataFrame, inferring the datatypes. Rows are constructed by passing a list of key/value pairs as kwargs to the Row class. The keys of this list define the column names of the table, *and the types are inferred by sampling the whole dataset*, similar to the inference that is performed on JSON files.
{quote}
Schema inference works as expected when reading delimited filed like
{code}
spark.read.format('csv').option('inferSchema', True)...
{code}
but not when using toDF() / createDataFrame() API calls.
Spark 2.2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org