[jira] [Created] (SPARK-23786) CSV schema validation - column names are not checked

["Maxim Gekk (JIRA)" <ji...@apache.org>]

Maxim Gekk created SPARK-23786:
----------------------------------

             Summary: CSV schema validation - column names are not checked
                 Key: SPARK-23786
                 URL: https://issues.apache.org/jira/browse/SPARK-23786
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.3.0
            Reporter: Maxim Gekk


Here is a csv file contains two columns of the same type:
{code}
$cat marina.csv
depth, temperature
10.2, 9.0
5.5, 12.3
{code}

If we define the schema with correct types but wrong column names (reversed order):
{code:scala}
val schema = new StructType().add("temperature", DoubleType).add("depth", DoubleType)
{code}

Spark reads the csv file without any errors:
{code:scala}
val ds = spark.read.schema(schema).option("header", "true").csv("marina.csv")
ds.show
{code}
and outputs wrong result:
{code}
+-----------+-----+
|temperature|depth|
+-----------+-----+
|       10.2|  9.0|
|        5.5| 12.3|
+-----------+-----+
{code}
The correct behavior would be either output error or read columns according its names in the schema.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org