[jira] [Resolved] (SPARK-18270) Users schema with non-nullable properties is overidden with true

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-18270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-18270.
----------------------------------
    Resolution: Duplicate

I am resolving this as a duplicate of SPARK-16472. Please reopen this if anyone thinks this is not the duplicate with some reasons.

> Users schema with non-nullable properties is overidden with true
> ----------------------------------------------------------------
>
>                 Key: SPARK-18270
>                 URL: https://issues.apache.org/jira/browse/SPARK-18270
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.0.1
>            Reporter: Jork Zijlstra
>
> Users schema with non-nullable properties is overidden with true in CSVRelation.csvParser.
> The schema that is given to the CSVRelation.csvParser(schema: StructType) isnt the version that is user specifies.
> All nullable option are set to true
> Specifying the schema:
> StructType(Array(
>   StructField("id", IntegerType, nullable = false),
>   StructField("underlyingId", IntegerType, true)
> ))
> Read the data:
> sparkSession.read
>     .schema(sourceSchema)
>     .option("header", "false")
>     .option("delimiter", """\t""")
>     .csv(files(dates): _*)
>     .rdd
> Actual Result: 
> schema inside csvParser contains only nullable = true values.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org