You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Wenchen Fan (JIRA)" <ji...@apache.org> on 2016/09/29 15:31:21 UTC

[jira] [Created] (SPARK-17725) Spark should not write out parquet files with schema containing non-nullable fields

Wenchen Fan created SPARK-17725:
-----------------------------------

             Summary: Spark should not write out parquet files with schema containing non-nullable fields
                 Key: SPARK-17725
                 URL: https://issues.apache.org/jira/browse/SPARK-17725
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.0
            Reporter: Wenchen Fan


Since Spark 1.3, after PR https://github.com/apache/spark/pull/4826 , Spark SQL will always set all schema fields to nullable before writing out parquet files, to make the data pipeline more robust.

However, this behaviour has been changed in 2.0 accidently by PR https://github.com/apache/spark/pull/11509



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org