[jira] [Comment Edited] (SPARK-22505) toDF() / createDataFrame() type inference doesn't work as expected

["Ruslan Dautkhanov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-22505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263733#comment-16263733 ] 

Ruslan Dautkhanov edited comment on SPARK-22505 at 10/5/18 8:26 PM:
--------------------------------------------------------------------

[~hyukjin.kwon], I tried to convert an rdd containg csv and it doesn't seem to infer data types correctly:

{code:python}rdd1 = sc.parallelize([('1','a'),('2','b'),('3','c')])
df = spark.read.csv(rdd1){code}

and 

{code:python}
rdd2 = sc.parallelize([('1,a'),('2,b'),('3,c')])
df = spark.read.csv(rdd2) 
{code}

- in both cases `df.printSchema()` print 

{noformat}
root
 |-- _c0: string (nullable = true)
 |-- _c1: string (nullable = true)
{noformat}

while the first column has to have `int` as inferred data type.


was (Author: tagar):
that's great. thank you [~hyukjin.kwon]

> toDF() / createDataFrame() type inference doesn't work as expected
> ------------------------------------------------------------------
>
>                 Key: SPARK-22505
>                 URL: https://issues.apache.org/jira/browse/SPARK-22505
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark, Spark Core
>    Affects Versions: 2.2.0, 2.3.0
>            Reporter: Ruslan Dautkhanov
>            Priority: Major
>              Labels: csvparser, inference, pyspark, schema, spark-sql
>
> {code}
> df = sc.parallelize([('1','a'),('2','b'),('3','c')]).toDF(['should_be_int','should_be_str'])
> df.printSchema()
> {code}
> produces
> {noformat}
> root
>  |-- should_be_int: string (nullable = true)
>  |-- should_be_str: string (nullable = true)
> {noformat}
> Notice `should_be_int` has `string` datatype, according to documentation:
> https://spark.apache.org/docs/latest/sql-programming-guide.html#inferring-the-schema-using-reflection
> {quote}
> Spark SQL can convert an RDD of Row objects to a DataFrame, inferring the datatypes. Rows are constructed by passing a list of key/value pairs as kwargs to the Row class. The keys of this list define the column names of the table, *and the types are inferred by sampling the whole dataset*, similar to the inference that is performed on JSON files.
> {quote}
> Schema inference works as expected when reading delimited files like
> {code}
> spark.read.format('csv').option('inferSchema', True)...
> {code}
> but not when using toDF() / createDataFrame() API calls.
> Spark 2.2.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org