You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Harry Brundage (JIRA)" <ji...@apache.org> on 2015/11/19 19:31:10 UTC
[jira] [Comment Edited] (SPARK-11319) PySpark silently Accepts null
values in non-nullable DataFrame fields.
[ https://issues.apache.org/jira/browse/SPARK-11319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014098#comment-15014098 ]
Harry Brundage edited comment on SPARK-11319 at 11/19/15 6:30 PM:
------------------------------------------------------------------
Forgive my frankness but that is ridiculous. This means anyone reading a JSON or CSV file needs to do their own validation pass over the data before passing it to SparkSQL. For anyone working with any kind of datasource they don't trust completely, this renders the entire loader layer of SparkSQL useless, and forces each user to implement their own. You have an opportunity to solve your users problems of notifying them when their schema expectations are violated and instead you silently allow and happily encourage data quality issues. Every database ever validates data on input!
was (Author: airhorns):
Forgive my frankness but that is ridiculous. This means anyone reading a JSON or CSV file needs to do their own validation pass over the data before passing it to SparkSQL. For anyone working with any kind of datasource they don't trust completely, this renders the entire loader layer of SparkSQL useless, and forces each user to implement their own. You have an opportunity to solve your users problems of notifying them when their schema expectations are violated and instead you silently allow and happily encourage data quality issues. Every database worth its salt validates data on input!
> PySpark silently Accepts null values in non-nullable DataFrame fields.
> ----------------------------------------------------------------------
>
> Key: SPARK-11319
> URL: https://issues.apache.org/jira/browse/SPARK-11319
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Reporter: Kevin Cox
>
> Running the following code with a null value in a non-nullable column silently works. This makes the code incredibly hard to trust.
> {code}
> In [2]: from pyspark.sql.types import *
> In [3]: sqlContext.createDataFrame([(None,)], StructType([StructField("a", TimestampType(), False)])).collect()
> Out[3]: [Row(a=None)]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org